:W&Bá¾D|ëM rmg1jlqkª; 9 /0.28.1+bc2cf24b1bd9fa3a1b6cbeb920087b832afc526f0.70.0•†´CÂ" 1fqt0rwnkxbv rmg1jlqkŠŸ rmg1jlqkspatop" _wandb‚{}Bspatop_v6_blocks_outputjdp-spatop-train-v6-blocks-4gd2cŠ ­–ÄÓˆÿ§š1 (H¨¸h"3.12.10*0.28.1j linux-x86_64ö'jÊcÒÝ -Linux-6.8.0-134-generic-x86_64-with-glibc2.36CPython 3.12.10 ¬–ÄÓè”°«*-of*A/data/spatop/options_files/sweep_fixed_dp/config_gs8pex8v_v6.json*-l*/scratch/logs/*-n*spatop_v6_blocks_output2-m spanet.trainRdanielprimosch@gmail.comZ/scratch/logs/bdp-spatop-train-v6-blocks-4gd2cr/opt/conda/bin/pythonº 4x78cejvw03lw1oocuycucnzl5165t9u79ëŸÇÒÁ€@ˆ@’NVIDIA A100 80GB PCIe˜¢ /€€€´ð7€Àʪ骀À½­ÝÂS NVIDIA A100 80GB PCIe€€€€À€6"Ampere*(GPU-1c79f124-b049-1e3d-7f6d-104357f4ddb6Ê13.2º 4x78cejvw03lw1oocuycucnzl5165t9uæPò2  requirements.txt°µ…ÄP rmg1jlqkZ? (H¨¸ (H¨¸h"3.12.10*0.28.1j linux-x86_64rz{R rmg1jlqkZA (H¨¸ (H¨¸8h"3.12.10*0.28.1j linux-x86_64SM/* rmg1jlqkb trainer/global_step2Óµ‹šU  rmg1jlqkZD (H¨¸ (H¨¸8h"3.12.10*0.28.1j linux-x86_64)õ/  rmg1jlqkb*"trainer/global_step2VÔrmK  rmg1jlqkj: ­–ÄÓ˜¶ÝÜ*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] †0ÿ’  rmg1jlqkj€  ®–ÄÓ˜žÚ?î â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 220 K │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 332 K │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 4.8 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 12 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ ¶7±Ç[  rmg1jlqkjÉ ®–ÄÓ°›ÿ@·Trainable params: 5.3 M Non-trainable params: 50 Total params: 5.3 M Total estimated model params size (MB): 21.374 Modules in train mode: 2326 Modules in eval mode: 0 Total FLOPs: 0 ›Ê@½ rmg1jlqk*«  hidden_dim‚64  transformer_dim‚64  transformer_dim_scale‚2.0  initial_embedding_dim‚64  position_embedding_dim‚64  num_embedding_layers‚10  num_encoder_layers‚4 ! num_branch_embedding_layers‚3  num_branch_encoder_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1  num_attention_heads‚8 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 K event_info_file‚7"/data/spatop/event_files/v11/tt_hadronic_v7_full.yaml" K training_file‚9"/data/spatop/delphes_v6/tt_hadronic_training2k_clean.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_limit‚1.0  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  batch_size‚512  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW" ' learning_rate‚0.0007344299982717679  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0  dropout‚0.2978723651141102 epochs‚10 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"ùn)oê rmg1jlqkjØ ®–ÄÓЦÊFÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ãx_M rmg1jlqkj» ®–ÄÓÈ™ëF«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=7` in the `DataLoader` to improve performance. æT7oë rmg1jlqkjÙ °–ÄÓˆÛ­„È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ÕÐé¸P rmg1jlqkj¾ °–ÄÓ¸†Ä„­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=7` in the `DataLoader` to improve performance. Kã*¦Ž‚(:„ ¼–ÄÓ™ò¤¹ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.979' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚30.993 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07¬ù‘,‚(:¢  ¼–ÄÓÜØ–» disk.rbd1.in‚0 disk.rbd1.out‚0 disk.drbd1552.in‚0 cpu‚18.674445919106297 disk.drbd1289.in‚0 disk.drbd1550.out‚0 network.sent‚43080 disk.sda.out‚ 0.47265625 disk.drbd1673.in‚0 disk.drbd1226.in‚0 disk.drbd1817.out‚0$ proc.memory.rssMB‚11253.18359375 disk.drbd1965.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.nvme0n1.in‚0$ disk./.usageGB‚570.3252258300781 disk.drbd2078.in‚0 disk.drbd1550.in‚0 disk.sda.in‚0 disk.drbd1226.out‚0* disk./.usagePercent‚31.895572253328208 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1584.out‚0 disk.dm-0.out‚ 0.32421875$ memory_percent‚41.19939208030701* proc.memory.availableMB‚38535.56640625 disk.drbd2118.out‚0 disk.drbd1289.out‚0 network.recv‚45448 disk.drbd2041.out‚0 disk.dm-0.in‚0 disk.drbd1510.in‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.drbd1965.out‚0 disk.sda2.in‚0 disk.drbd1931.out‚0 disk.drbd1510.out‚0 disk.drbd1828.out‚0 disk.drbd1931.in‚0 disk.drbd1817.in‚0 disk.nvme0n1.out‚ 0.32421875 disk.drbd1356.in‚0 disk.sda2.out‚ 0.47265625 disk.drbd1673.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0* proc.memory.percent‚17.170995473861694 proc.cpu.threads‚29 disk.drbd1285.in‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.sda1.out‚0_BT3b/ lr-AdamW/pg2"trainer/global_step2:JPw€æn3b/ lr-AdamW/pg1"trainer/global_step2:JP?MQ#-) _wandbruntime‚25  _runtime‚25=Tä^À» ' lr-AdamW/pg2‚1.2628371377799987e-06 trainer/global_step‚49 # _timestamp‚1.78579335192247e+09 _runtime‚ 25.999836968 _step‚0 ' lr-AdamW/pg1‚1.2628371377799987e-06=ç˜Â½ _runtime‚ 25.999836968 _step‚0 ' lr-AdamW/pg1‚1.2628371377799987e-06 ' lr-AdamW/pg2‚1.2628371377799987e-06 trainer/global_step‚49 # _timestamp‚1.78579335192247e+09YÕjfbb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JPRЦsbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JP jºürbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPQ±ácb_ DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JP2l²_7b[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JPFɳïg8bc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JPF;#k9bg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JPVÐ~ãl:bh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JPJ)Œv;br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPG' šqbc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP;,a?b] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JP žOe@ba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPªŸªlAbh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPóÇ!|jBbf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JP¼²z‘kCbg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JPD¯IÚ_Db[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JPK!jEbf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JPDÃmjFbf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JPËwyjGbf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JP©ç^HbZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JPTm>kIbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JP¦ñnJbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP±0æmKbi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JP î  nLbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPë}ÄCiMbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JPÐ2±nNbj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JP£;ÝÃcOb_ DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPËâ‹gVbc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPëd†½gWbc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JP³€€pXbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPæéj[kYbg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JPtv[³jZbf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JP*“˜m[bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPnSv“p\bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JP’Åÿób]b^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JP·ý¡8p^bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPð•ük_bg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JPžÃŠ]q`bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPf>W]oabk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JP%k¥ígbbc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JPFÇgcbc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPÿ=ddb` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JP‘9lÓpebl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JPïd3ofbk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPr4Q¼tgbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPííïcshbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JP¨"«eiibe BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JP eStjbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP„^64`kb\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JP3Þ^clb_ DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JPÆ°“©mbh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JP ¿°oŽbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPñÉ1’ibd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JPI„3ibd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JP÷4Òëf‘ba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPM°Ž|o’bj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPÅO´Fo“bj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPv®–Bc”b^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JP]Rå«.•) _wandbruntime‚26  _runtime‚26â³E$–ÝN VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚144579 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 885141475884 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940454912 RADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 906456278528 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 862880830464 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚514 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 45048632832 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚871036 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 866205078528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _runtime‚ 26.001551958 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 [JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 911253711360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940454912 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 906560747520 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚524651 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚343826 C8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚861378 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚336729 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚534307 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11885 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250522112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886704 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚336729 GDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚343574 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚504 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 43679917056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11885 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 841553293472 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 44920055808 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 841570332832 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 866180174848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11814 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 885038633400 _step‚1 O>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 906456278528 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚534307 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚524651 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 43679917056 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allo¶ ]1Ócated‚ 2768240640 GDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚343574 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚504 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 43679917056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11885 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 841553293472 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 44920055808 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 841570332832 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 866180174848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11814 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 885038633400 _step‚1 O>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 906456278528 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚534307 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚524651 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 43679917056 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 GDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _runtime‚ 26.001551958 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 [JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 911253711360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940454912 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 906560747520 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚524651 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚343826 C8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚861378 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚336729 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚534307 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11885 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250522112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886704 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚336729MÍáe˜b` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JPfU"h™bc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JP¹¬ùsšbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP°Êq¨b›b] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JPx ×jmœbh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JP yÒjbe BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JP)>xŸižbd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JPîJhŸbc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JPÒTÍu bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JP›t6†w¡br ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPˆvÆa¢b\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JP˜pk£bf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JP ‰b¤b] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JPu¸ºp¥bk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JPØE+l¦bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JP5Uò`§b[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JP§n¥cg¨bb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JP¯‡Ÿg©bb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JP]8¥Hjªbe BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JP¼×›u«bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPu;õåV¬bQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JP¨UŠb­b] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JP9:ƒhf®ba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPáªA_¯bZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JP'E k°bf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JPd¶q¨l±bg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPIC0—_²bZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPÑyñÝg³bb ?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current"trainer/global_step2:JP¾‡n´bi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JP Žùhµbc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JPyBi¶bd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JP[Ò¯‘c·b^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JP7Eÿ!r¸bm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPe4y“l¹bg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JPMÓnXqºbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JPÔ`â‘`»b[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JPåVEj¼be BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JP¼ølpq½bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JP·Ì g]¾bX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPWÝ.Øk¿bf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPÓÉ`dÀb_ oÇbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP×~ERaÈb\ 9DeviceStatsMonitor.on_train_batch_end/segment.all.current"trainer/global_step2:JPøtfgÉbb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JPâSHmÊbh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPسÑ\ËbW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JPZx ÐrÌbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JPTl,pÍbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPnG÷kÎbf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JP´{ iÏbd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JPy{nÐbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JPŽ©¸VqÑbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JP¿”çíeÒb` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPo‰2çfÓba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JP’ÔâøkÔbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPóäknmÕbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPéÚGÚmÖbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JP!Qиi×bd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JP)•¢±mØbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPa3AeÙb` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JP~a ÛhÚbc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JPÛÓØNhÛbc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JPœ0‰šrÜbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JPkîÜfÝba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP÷<¼mÞbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPL3Ë~^ßbY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JP'«Û,fàba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP?‚Qmábh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPóø;¡lâbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JP[M‹3iãbd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JPZyýïpäbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JPKu×råbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP¨•h^qæbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPÛ­Kæjçbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JPyZXmèbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPtgBzoébj GDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current"trainer/global_step2:JP]»mêbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPÕô&îoëbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPì]¹‡lìbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JP™±þdíb_ /Q¨sóbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP¦+i»hôbc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JP¡TÜdõb_ DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JP’cÛŒoúbj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JPmlö>pûbk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JP@ųCeüb` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JP¢ “cýb^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JPɳŒgþbb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JPÑ¥—mÿbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JPjÖuÐi€bd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JP. Unbi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP\(ãl‚bg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JP³ñÕjƒbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JPd‚ f„ba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPÉCQd…b_ DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JP?!h‚^‘bY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JPwÃ’f’ba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPÑç ª.“) _wandbruntime‚26  _runtime‚26˜¢0h”èL JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚203184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 880030554112 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 858310505152 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚544844 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 44559543808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 TCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 902759121538 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 44362813150 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚147431 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 44472107008 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚203182 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 880047593472 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚888216 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚878544 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚147176 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 44448616386 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚504 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚350358 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 YHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 929364081664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚343370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147182080 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 % _timestamp‚1.7857933523007247e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚343370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 883387291648 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚878544 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚343372 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚49 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚544844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_poo>ŒÎl.allocated‚ 45951886336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚888216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 902656278942 _runtime‚ 26.378458613 P?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 924502661120 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚350615 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122278400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 929216899584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 858293465792 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚343372 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200*p§Zr&•ìL XGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 880047593472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚535174 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 44559543808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚343370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 883387291648 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚878544 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚343372 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚49 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚544844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 45951886336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚888216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 902656278942 _runtime‚ 26.378458613 P?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 924502661120 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚350615 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122278400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 929216899584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 858293465792 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚343372 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚203184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 880030554112 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 858310505152 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚544844 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 44559543808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 TCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 902759121538 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 44362813150 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚147431 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 44472107008 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚203182 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 880047593472 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚888216 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚878544 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚147176 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 44448616386 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚504 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚350358 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 YHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 929364081664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚343370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147182080 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 % _timestamp‚1.7857933523007247e+09 F loss/SRqqt1/assignment_loss"trainer/global_step2:JPïÛC¢b> loss/SRqqt2/assignment_loss"trainer/global_step2:JP{ 1A£b< loss/FBt1/assignment_loss"trainer/global_step2:JPe×^<.¤) _wandbruntime‚26  _runtime‚26 W¦„ø¥ò /loss/FBt2/assignment_loss‚2.281000092625618 .loss/FRt1/detection_loss‚3.824740376586866 .loss/FRt2/detection_loss‚5.129910856485367 1loss/SRqqt2/detection_loss‚2.9626637726264486 _step‚3 0loss/FRt2/assignment_loss‚5.8206438512513134 1loss/SRqqt1/detection_loss‚4.2221398293262435 /loss/FBt1/detection_loss‚21.754089613171185 /loss/FBt2/detection_loss‚16.265543619791668 %loss/total_loss‚79.49345066574585 epoch‚0 1loss/SRqqt1/assignment_loss‚4.574840105715252 1loss/SRqqt2/assignment_loss‚4.543146046541505 0loss/FBt1/assignment_loss‚2.3697859399458943 trainer/global_step‚49 % _timestamp‚1.7857933523027976e+09 _runtime‚ 26.37969869 /loss/FRt1/assignment_loss‚5.744946561678492Ïü¦ö /loss/FBt1/detection_loss‚21.754089613171185 /loss/FBt2/detection_loss‚16.265543619791668 %loss/total_loss‚79.49345066574585 epoch‚0 1loss/SRqqt1/assignment_loss‚4.574840105715252 1loss/SRqqt2/assignment_loss‚4.543146046541505 0loss/FBt1/assignment_loss‚2.3697859399458943 trainer/global_step‚49 % _timestamp‚1.7857933523027976e+09 _runtime‚ 26.37969869 /loss/FRt1/assignment_loss‚5.744946561678492 /loss/FBt2/assignment_loss‚2.281000092625618 .loss/FRt1/detection_loss‚3.824740376586866 .loss/FRt2/detection_loss‚5.129910856485367 1loss/SRqqt2/detection_loss‚2.9626637726264486 _step‚3 0loss/FRt2/assignment_loss‚5.8206438512513134 1loss/SRqqt1/detection_loss‚4.2221398293262435œ[0‚œ§‚(:‘ Ë–ÄÓÞ毹 gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.52' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.839999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QÔÎï5¨‚(:ª  Ë–ÄÓÓùɹ$ disk./.usageGB‚570.3253021240234 disk.drbd1965.out‚0 disk.rbd1.out‚ 0.0322265625% memory_percent‚41.200220584869385* disk./.usagePercent‚31.895576520085022 disk.drbd1356.in‚0 disk.drbd1673.out‚0 disk.drbd1552.out‚0 disk.drbd1828.out‚0 disk.drbd1931.out‚0 disk.drbd1510.in‚0 disk.drbd2078.in‚0 disk.drbd1931.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 network.recv‚46969 disk.rbd0.out‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.sda.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd2078.out‚0! proc.memory.rssMB‚ 11253.21875 proc.cpu.threads‚29 disk.drbd1938.in‚0 disk.drbd1285.out‚0 disk.sda2.out‚0.828125 disk.drbd2111.out‚0 disk.drbd2041.in‚0* proc.memory.percent‚17.171049118041992 disk.nvme0n1.in‚0 disk.drbd1356.out‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.dm-0.in‚0! disk.nvme0n1.out‚ 0.8740234375 disk.drbd1965.in‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.drbd1285.in‚0 network.sent‚50271 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.sda.out‚0.828125 disk.sda1.in‚0 disk.dm-0.out‚ 0.8740234375 disk.drbd1673.in‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0) proc.memory.availableMB‚ 38535.0234375 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.drbd2111.in‚0 cpu‚17.879112774982424¤‡?·5©‚(:ª  Ú–ÄÓÏÚ½¹* disk./.usagePercent‚31.895575453395818$ disk./.usageGB‚570.3252830505371! disk.nvme0n1.out‚ 1.0888671875 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1552.out‚0 disk.drbd1285.out‚0 disk.drbd1289.out‚0 disk.sda.in‚0 disk.drbd1226.in‚0 disk.rbd0.out‚0 disk.drbd1226.out‚0 disk.drbd1285.in‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0* proc.memory.percent‚17.171049118041992 network.sent‚101097 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1584.in‚0 network.recv‚51761 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1938.in‚0 disk.drbd1673.out‚0 disk.drbd1817.in‚0 proc.cpu.threads‚29 disk.drbd1356.in‚0 disk.rbd1.out‚ 0.0322265625 disk.sda2.in‚0 disk.sda2.out‚1.15625 disk.drbd2041.in‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.dm-0.out‚ 1.0888671875* proc.memory.availableMB‚38535.35546875 disk.drbd2111.out‚0 disk.sda1.in‚0 cpu‚17.302406304807068 disk.drbd1550.out‚0 disk.sda1.out‚0% memory_percent‚41.199713945388794 disk.drbd1931.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0! proc.memory.rssMB‚ 11253.21875 disk.drbd2118.in‚0 disk.drbd2041.out‚0 disk.dm-0.in‚0 disk.nvme0n1.in‚0 disk.rbd0.in‚0 disk.drbd1550.in‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.sda.out‚1.15625 disk.drbd1673.in‚0 disk.drbd1817.out‚0“ÀT‹›ª‚(: Ú–ÄÓªŒõ¹ gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚33.0 gpu.0.powerWatts‚97.132' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚32.37733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿À«É.«) _wandbruntime‚48  _runtime‚48H*í¬¼ trainer/global_step‚99 $ _timestamp‚1.785793374570321e+09 _runtime‚ 48.647278671 _step‚4 ' lr-AdamW/pg1‚2.5514464620453036e-06 ' lr-AdamW/pg2‚2.5514464620453036e-066å#=Æ­À _step‚4 ' lr-AdamW/pg1‚2.5514464620453036e-06 ' lr-AdamW/pg2‚2.5514464620453036e-06 trainer/global_step‚99 $ _timestamp‚1.785793374570321e+09 _runtime‚ 48.647278671Cö‰.®) _wandbruntime‚48  _runtime‚48"›!ö|'¯öN G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 _runtime‚ 48.648663549 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1052653 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 1808790564352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 1721218980864 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 90116208640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 87373803436 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 1726560918528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚668877 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚287303 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 1765935712908 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 87588622848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚396097 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658çøÜ(°úN O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 1721201941504 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 87676052480 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚683400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 1678578948832 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 87676052480 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1731188 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7857933745713723e+09 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 1808790564352 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1062309 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚683141 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 1816811995648 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1721530 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 1726536014848 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 87459606560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1062309 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1052653 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.aŨ¾Xll.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 _runtime‚ 48.648663549 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1052653 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 1808790564352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 1721218980864 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 90116208640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 87373803436 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 1726560918528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚668877 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚287303 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 1765935712908 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 87588622848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚396097 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 H6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚675522 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚400034 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 1826941423104 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚675520 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 1826836946944 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚400032 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1072846 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 1743743131648 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 91151332352 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 91018568192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 1738385743872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 1738385743872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 1743768035328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 1783553358450 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 88555679232 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1748368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1695302081792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 1826941423104 $ _timestamp‚1.785793374947142e+09 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1738696 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚689957 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 1783656201046 _runtime‚ 49.024516236 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 88337079894 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚690200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1834761699840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 88468242432 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚675522 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 _step‚6 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1738696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1748368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840F·1Á‰&³ƒM [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 91018568192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 1738385743872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 1738385743872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 1743768035328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 1783553358450 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 88555679232 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1748368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1695302081792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 1826941423104 $ _timestamp‚1.785793374947142e+09 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1738696 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚689957 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 1783656201046 _runtime‚ 49.024516236 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 88337079894 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚690200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1834761699840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 88468242432 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚675522 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 _step‚6 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1738696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1748368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚290166 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚289925 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1072846 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚675522 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚400034 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 1826941423104 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚675520 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 1826836946944 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚400032 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1072846 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 1743743131648 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 91151332352GaÔ¨.´) _wandbruntime‚49  _runtime‚49cÓ5ƒøµò _step‚7 /loss/FRt2/assignment_loss‚5.849792473752734 1loss/SRqqt2/assignment_loss‚4.539884092346314 0loss/FBt2/assignment_loss‚2.2514979010042935 /loss/FBt1/detection_loss‚30.821101747453213 /loss/FBt2/detection_loss‚16.955709631028384 0loss/FRt1/assignment_loss‚5.7297220069370915 1loss/SRqqt1/detection_loss‚3.5704862890821514 %loss/total_loss‚88.29018367620928 trainer/global_step‚99 _runtime‚ 49.025773758 1loss/SRqqt1/assignment_loss‚4.484711175013071 0loss/FBt1/assignment_loss‚2.3181225856145224 .loss/FRt1/detection_loss‚4.252322071723723 -loss/FRt2/detection_loss‚4.74444724556426 0loss/SRqqt2/detection_loss‚2.772386456689527 epoch‚0 % _timestamp‚1.7857933749489737e+09Zwªü¶ö /loss/FRt2/assignment_loss‚5.849792473752734 1loss/SRqqt2/assignment_loss‚4.539884092346314 0loss/FBt2/assignment_loss‚2.2514979010042935 /loss/FBt1/detection_loss‚30.821101747453213 /loss/FBt2/detection_loss‚16.955709631028384 0loss/FRt1/assignment_loss‚5.7297220069370915 1loss/SRqqt1/detection_loss‚3.5704862890821514 %loss/total_loss‚88.29018367620928 trainer/global_step‚99 _runtime‚ 49.025773758 1loss/SRqqt1/assignment_loss‚4.484711175013071 0loss/FBt1/assignment_loss‚2.3181225856145224 .loss/FRt1/detection_loss‚4.252322071723723 -loss/FRt2/detection_loss‚4.74444724556426 0loss/SRqqt2/detection_loss‚2.772386456689527 epoch‚0 % _timestamp‚1.7857933749489737e+09 _step‚7—仹œ·‚(:‘ é–ÄÓöš†¹ gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚34.0 gpu.0.powerWatts‚128.024' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚42.67466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óº>¸‚(:³  é–ÄÓıӹ* disk./.usagePercent‚31.895578866801273 disk.drbd1550.out‚0 disk.drbd2111.in‚0 disk.sda.out‚ 1.55859375 disk.dm-0.in‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0" disk.nvme0n1.out‚ 3.24658203125 network.sent‚104316 disk.drbd1356.out‚0 disk.drbd1552.in‚0* proc.memory.availableMB‚38534.66015625 disk.rbd1.out‚ 0.09521484375 disk.drbd2118.out‚0 disk.drbd1510.out‚0 disk.sda2.in‚0 disk.drbd1289.out‚0 disk.drbd2118.in‚0 disk.drbd1584.out‚0 disk.drbd1285.in‚0 disk.drbd1965.in‚0 disk.drbd1584.in‚0 disk.drbd1965.out‚0 disk.drbd1550.in‚0 disk.nvme0n1.in‚0 disk.sda.in‚0 disk.drbd1931.out‚0 disk.drbd1285.out‚0 disk.drbd1828.in‚0 disk.drbd1817.out‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.sda1.out‚0 disk.drbd1931.in‚0! proc.memory.rssMB‚ 11253.21875 disk.sda2.out‚ 1.55859375 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.drbd2041.in‚0 disk.dm-0.out‚ 3.24658203125 disk.drbd1673.out‚0 disk.drbd1356.in‚0 disk.rbd0.in‚0 disk.drbd1938.out‚0 disk.drbd1510.in‚0% memory_percent‚41.200774908065796 network.recv‚52909 disk.rbd0.out‚0 disk.drbd1828.out‚0 disk.drbd1673.in‚0 disk.drbd1226.in‚0 disk.drbd2078.in‚0 proc.cpu.threads‚29 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.drbd1552.out‚0* proc.memory.percent‚17.171049118041992 cpu‚16.851297049107153$ disk./.usageGB‚570.3253440856934 disk.rbd1.in‚0 @že.¹) _wandbruntime‚71  _runtime‚71Âï7º¼ trainer/global_step‚149 % _timestamp‚1.7857933970579028e+09 _runtime‚ 71.136666046 _step‚8 & lr-AdamW/pg1‚3.840055786310608e-06 & lr-AdamW/pg2‚3.840055786310608e-06wö”<Æ»À _step‚8 & lr-AdamW/pg1‚3.840055786310608e-06 & lr-AdamW/pg2‚3.840055786310608e-06 trainer/global_step‚149 % _timestamp‚1.7857933970579028e+09 _runtime‚ 71.136666046´ÃÃ%.¼) _wandbruntime‚71  _runtime‚71&óÀ* ½…O HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1580655 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 131348070068 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚149 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2711124850176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚430122 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 135471309824 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 2515570525472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 131672187904 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 2646832792416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 131262266944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 2579557131264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1023067•Ÿ '¾‰O % _timestamp‚1.7857933970586567e+09 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2581682 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 2579540091904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 131584758272 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 131672187904 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 131584758272 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚592947 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 2586891854848 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 2579540091904 _step‚9 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1001027 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1001027 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 2722388068352 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1580655 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 131348070068 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚149 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2711124850176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚430122 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 135471309824 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 2515570525472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 131672187904 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 2646832792416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 131262266944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 2579557131264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1023067 HDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1030145 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 2604098971648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1591178 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2729275708928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1007672 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 2729171232768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 132464377856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 132225543402 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 132551814656 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1600848 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 2596706854912 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 2740344391680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚596882 N6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 # _timestamp‚1.78579339743753e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚ 71.514988511 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2596706854912 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 136245420032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 2664553280554 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2598848 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚522 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2729275708928 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚596884 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 132551814656 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 2740495768064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2598848 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2608520 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2608520 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1007672 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1591178 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 136371892736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 2532310697792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 _step‚10 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1029887&8ª?+Á’M TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚596884 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 132551814656 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 2740495768064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2598848 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2608520 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2608520 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1007672 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1591178 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 136371892736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 2532310697792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 _step‚10 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1029887 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 2604123875328 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 2664450437958 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1030145 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 2604098971648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1591178 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2729275708928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1007672 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 2729171232768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStat9[iñmsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 132464377856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 132225543402 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 132551814656 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1600848 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 2596706854912 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 2740344391680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚596882 N6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 # _timestamp‚1.78579339743753e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚ 71.514988511 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2596706854912 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 136245420032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 2664553280554 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2598848 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚522 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2729275708928 ÓÔë.Â) _wandbruntime‚71  _runtime‚71%·SööÃð /loss/FRt2/assignment_loss‚5.689925892712319 1loss/SRqqt2/assignment_loss‚4.591133949707966 .loss/FRt2/detection_loss‚4.559833316974444 .loss/FBt1/detection_loss‚21.49834113261279 /loss/FBt2/detection_loss‚16.124168513342738 # _timestamp‚1.78579339743946e+09 _step‚11 0loss/FBt2/assignment_loss‚2.2716694275538125 .loss/FRt1/detection_loss‚4.101071149110794 0loss/SRqqt1/detection_loss‚4.237306230678791 /loss/FRt1/assignment_loss‚5.658464641981228 1loss/SRqqt1/assignment_loss‚4.521608890556708 0loss/FBt1/assignment_loss‚2.2724880470949063 0loss/SRqqt2/detection_loss‚3.509464829247825 %loss/total_loss‚79.03547602157433 epoch‚0 trainer/global_step‚149 _runtime‚ 71.516256377-uÄúÄô 1loss/SRqqt1/assignment_loss‚4.521608890556708 0loss/FBt1/assignment_loss‚2.2724880470949063 0loss/SRqqt2/detection_loss‚3.509464829247825 %loss/total_loss‚79.03547602157433 epoch‚0 trainer/global_step‚149 _runtime‚ 71.516256377 /loss/FRt2/assignment_loss‚5.689925892712319 1loss/SRqqt2/assignment_loss‚4.591133949707966 .loss/FRt2/detection_loss‚4.559833316974444 .loss/FBt1/detection_loss‚21.49834113261279 /loss/FBt2/detection_loss‚16.124168513342738 # _timestamp‚1.78579339743946e+09 _step‚11 0loss/FBt2/assignment_loss‚2.2716694275538125 .loss/FRt1/detection_loss‚4.101071149110794 0loss/SRqqt1/detection_loss‚4.237306230678791 /loss/FRt1/assignment_loss‚5.658464641981228 Fë¤ï‘Å‚(:† ø–ÄÓ™¢ˆ¹ gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.826' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.942 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0túJ©=Æ‚(:²  ø–ÄÓ캹 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.sda.out‚ 2.23046875 disk.dm-0.in‚0! proc.memory.rssMB‚ 11253.21875 network.sent‚149214 disk.drbd1285.in‚0 disk.drbd1828.in‚0 network.recv‚55653 disk.drbd2111.out‚0 disk.drbd2041.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1938.in‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.sda2.in‚0 disk.drbd1584.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 proc.cpu.threads‚29 disk.drbd1965.in‚0 disk.drbd1289.out‚0* proc.memory.percent‚17.171049118041992 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.rbd0.out‚0 disk.drbd2111.in‚0 disk.drbd1584.out‚0 disk.sda.in‚0$ memory_percent‚41.20197892189026 disk.drbd1356.in‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1510.in‚0" disk.nvme0n1.out‚ 3.37939453125 disk.drbd2078.out‚0 disk.dm-0.out‚ 3.37939453125 disk.drbd2041.out‚0 disk.drbd1817.out‚0 disk.nvme0n1.in‚0 disk.drbd1285.out‚0 disk.rbd1.out‚ 0.09521484375 disk.drbd1931.out‚0 disk.drbd1938.out‚0* proc.memory.availableMB‚38533.87109375 cpu‚16.501942825399055* disk./.usagePercent‚31.895586760301388 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.drbd1817.in‚0 disk.drbd1510.out‚0$ disk./.usageGB‚570.3254852294922 disk.sda2.out‚ 2.23046875 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd1550.out‚0 disk.drbd1673.out‚0~ÐlœÇ‚(:‘ ‡—ÄÓ¥¥ò¸ gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚35.0 gpu.0.powerWatts‚100.867' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚33.62233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KªÑ:È‚(:¯  ‡—ÄÓ¹†¾¹ disk.drbd1931.in‚0 disk.drbd1817.in‚0 disk.drbd1552.in‚0* disk./.usagePercent‚31.895587186977064 disk.drbd2041.in‚0$ disk./.usageGB‚570.3254928588867 disk.rbd0.out‚0 disk.sda2.out‚ 2.52734375 disk.sda.out‚ 2.52734375! disk.nvme0n1.out‚ 4.2685546875 disk.drbd1289.in‚0 disk.drbd1550.out‚0 disk.drbd1226.in‚0 disk.drbd1965.in‚0 cpu‚16.217017236783292 proc.cpu.threads‚29 disk.drbd1828.in‚0 disk.sda.in‚0 disk.dm-0.in‚0 disk.drbd1938.out‚0 disk.drbd1285.out‚0! proc.memory.rssMB‚ 11253.21875* proc.memory.percent‚17.171049118041992 disk.drbd2111.in‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1356.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.rbd1.in‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd1673.out‚0 disk.drbd1510.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd1584.in‚0 disk.drbd1226.out‚0 disk.drbd1817.out‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd1552.out‚0 disk.drbd1965.out‚0 network.recv‚58312 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚0 disk.drbd1356.out‚0% memory_percent‚41.203147172927856 disk.drbd2111.out‚0 disk.rbd1.out‚ 0.158203125 network.sent‚194151 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.dm-0.out‚ 4.2685546875 disk.drbd2078.out‚0 disk.drbd1938.in‚0* proc.memory.availableMB‚38533.10546875VÝ).É) _wandbruntime‚93  _runtime‚93J¤ÿÃʽ & lr-AdamW/pg1‚5.128665110575913e-06 & lr-AdamW/pg2‚5.128665110575913e-06 trainer/global_step‚199 % _timestamp‚1.7857934195778337e+09 _runtime‚ 93.654811534 _step‚12çûʘÇËÁ _step‚12 & lr-AdamW/pg1‚5.128665110575913e-06 & lr-AdamW/pg2‚5.128665110575913e-06 trainer/global_step‚199 % _timestamp‚1.7857934195778337e+09 _runtime‚ 93.654811534 隀i.Ì) _wandbruntime‚93  _runtime‚93pÉŸÍŒ'͆O WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 3613563604992 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3451492 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3352596180832 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3627890563072 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 175150730452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _step‚13 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1333179 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 175580893696 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3627730790912 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 180617964544 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚573582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2118313 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3447272598528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 3437895281664 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1333179 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚526 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚789797 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3441834 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 175580893696 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3527832714408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 _runtime‚ 93.656184922 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2108657 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3437895281664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3613459136000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 3613563604992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 175236533576 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2108657 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 trainer/global_step‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320=%Æ¢'ΊO WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 3613563604992 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3451492 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3352596180832 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3627890563072 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 175150730452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _step‚13 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1333179 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 175580893696 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3627730790912 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 180617964544 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚573582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2118313 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3447272598528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 3437895281664 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1333179 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚526 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚789797 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3441834 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 175580893696 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3527832714408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 _runtime‚ 93.656184922 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2108657 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3437895281664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3613459136000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 3613563604992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 175236533576 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2108657 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 trainer/global_step‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 »{).Ï) _wandbruntime‚94  _runtime‚94W×ù1ÐM PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚793732 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3468672 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2119180 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3455062044672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3464479715328 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3369319313792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pooÆ×ÈÎd l.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚576397 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3645995225600 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2128850 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 3631609994752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 3455062044672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3369336353152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 176460513280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 3545347517466 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 176547950080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1339822 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1370131 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 176460513280 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3459000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1369888 _step‚14 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2119180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 176028203674 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3459000 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1339822 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 3545450360062 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 3631609994752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 94.028781988 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚516 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3468672 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2119180 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3455062044672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3464479715328 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3369319313792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚576397 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3645995225600 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2128850 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 3631609994752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 3455062044672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3369336353152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 176460513280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 3545347517466 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 176547950080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1339822 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1370131 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 176460513280 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3459000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1369888 _step‚14 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2119180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 176028203674 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3459000 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1339822 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 3545450360062 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 3631609994752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 94.028781988 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚516 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251í vL.Ò) _wandbruntime‚94  _runtime‚94R¨NëûÓõ epoch‚0 /loss/FRt1/assignment_loss‚5.823789100835819 0loss/FRt2/assignment_loss‚5.8253540595372515 .loss/FRt2/detection_loss‚5.553957518935204 0loss/SRqqt2/detection_loss‚2.986719158211989 _runtime‚ 94.030011234 1loss/SRqqt1/assignment_loss‚4.554412296840122 /loss/FBt2/detection_loss‚15.464747414588928 trainer/global_step‚199 /loss/FBt1/detection_loss‚24.332152452071508 %loss/total_loss‚81.29298225183072 2loss/SRqqt2/assignment_loss‚4.4776512788874765 0loss/FBt2/assignment_loss‚2.2763214015960695 /loss/FRt1/detection_loss‚3.6749757280444153 /loss/SRqqt1/detection_loss‚3.99992925851118 % _timestamp‚1.7857934199532413e+09 _step‚15 0loss/FBt1/assignment_loss‚2.3229725837707518ªÈ¢ÿÔù _runtime‚ 94.030011234 1loss/SRqqt1/assignment_loss‚4.554412296840122 /loss/FBt2/detection_loss‚15.464747414588928 trainer/global_step‚199 /loss/FBt1/detection_loss‚24.332152452071508 %loss/total_loss‚81.29298225183072 2loss/SRqqt2/assignment_loss‚4.4776512788874765 0loss/FBt2/assignment_loss‚2.2763214015960695 /loss/FRt1/detection_loss‚3.6749757280444153 /loss/SRqqt1/detection_loss‚3.99992925851118 % _timestamp‚1.7857934199532413e+09 _step‚15 0loss/FBt1/assignment_loss‚2.3229725837707518 epoch‚0 /loss/FRt1/assignment_loss‚5.823789100835819 0loss/FRt2/assignment_loss‚5.8253540595372515 .loss/FRt2/detection_loss‚5.553957518935204 0loss/SRqqt2/detection_loss‚2.986719158211989ßTeÕ‚(:„ –—ÄÓ»§ý¸ gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚36.0 gpu.0.powerWatts‚129.24' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚43.08 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž,{µ8Ö‚(:­  –—ÄÓ£À¹ disk.drbd1931.out‚0* proc.memory.availableMB‚38533.51953125) disk./.usagePercent‚31.89558612028786 disk.drbd1510.out‚0 disk.sda2.out‚ 2.88671875 disk.drbd2111.in‚0 disk.sda.in‚0! proc.memory.rssMB‚ 11253.21875 disk.drbd1938.in‚0 disk.nvme0n1.in‚0 disk.drbd1285.out‚0 network.sent‚197443 network.recv‚59621 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.drbd1226.out‚0 disk.drbd1356.out‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd2041.out‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.drbd1817.in‚0! disk.nvme0n1.out‚ 4.4912109375 disk.drbd1289.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd2111.out‚0 disk.drbd1584.out‚0 cpu‚15.981138552674626 disk.sda.out‚ 2.88671875 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd1828.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.rbd0.out‚0 disk.drbd1673.in‚0 disk.drbd2041.in‚0* proc.memory.percent‚17.171049118041992 disk.rbd1.in‚0 disk.drbd2078.out‚0 disk.sda1.in‚0 disk.dm-0.out‚ 4.4912109375 disk.drbd1673.out‚0 disk.drbd1226.in‚0$ memory_percent‚41.20251536369324 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1931.in‚0 disk.dm-0.in‚0 disk.rbd1.out‚ 0.158203125 disk.drbd1584.in‚0 disk.drbd2118.in‚0 proc.cpu.threads‚29$ disk./.usageGB‚570.3254737854004 disk.sda1.out‚0Zg¸î0×+ _wandbruntime‚115  _runtime‚115“Àð|Äؾ % _timestamp‚1.7857934417205079e+09 _runtime‚ 115.797488857 _step‚16 & lr-AdamW/pg1‚6.417274434841219e-06 & lr-AdamW/pg2‚6.417274434841219e-06 trainer/global_step‚249¦ši\ÈÙ _step‚16 & lr-AdamW/pg1‚6.417274434841219e-06 & lr-AdamW/pg2‚6.417274434841219e-06 trainer/global_step‚249 % _timestamp‚1.7857934417205079e+09 _runtime‚ 115.797488857@~0Ú+ _wandbruntime‚115  _runtime‚115!û(hŒ'Û†O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 4296216392704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 4515793421824 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _step‚17 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4533391397888 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 4189604796832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2636659 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4533235820032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4307603534848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 4408729793916 trainer/global_step‚249 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4301986 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 4515897890816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 4515897890816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2646315 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚517 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1665327 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 219664458752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚716623 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 _runtime‚ 115.798844212 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2646315 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 219577029120 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4311644 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 4296233432064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4307628438528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4515793421824 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1703268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1665327 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 225632285184ä÷AÜŠO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2646315 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚517 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDevice @InO&StatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1665327 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 219664458752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚716623 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 _runtime‚ 115.798844212 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2646315 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 219577029120 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4311644 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 4296233432064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4307628438528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4515793421824 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1703268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1665327 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 225632285184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 4296216392704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 4515793421824 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _step‚17 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4533391397888 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 4189604796832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2636659 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4533235820032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4307603534848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 4408729793916 trainer/global_step‚249 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4301986 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 4515897890816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 4515897890816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0)ñ?û0Ý+ _wandbruntime‚116  _runtime‚116ÂS•—&Þ‘M HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 220002470418 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1710047 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 220456648704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 4313400195072 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 4313400195072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 trainer/global_step‚249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 4313383155712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 4206327929792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 226665532928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 4206344969152 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4328824 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚527 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 4313383155712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2647182 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 220544085504 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4319152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4324835555328 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1671970 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4328824 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚990584 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚719708 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 220544085504 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1671972 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚990582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4319152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1710292 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1671972 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 4533944280576 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1671970 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 220456648704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 219916667182 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4551501088256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2656852 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4533839804416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 % _timestamp‚1.7857934420894794e+09 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4324810651648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251Ù8d›&ß•M E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚527 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 4313383155712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2647182 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 220544085504 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4319152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4324835555328 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1671970 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4328824 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚990584 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚719708 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 220544085504 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1671972 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚990582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4319152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1710292 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1671972 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 4533944280576 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1671970 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 220456648704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 219916667182 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4551501088256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2656852 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4533839804416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 % _timestamp‚1.7857934420894794e+09 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4324810651648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 220002470418 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1710047 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 220456648704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 4313400195072 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 4313400195072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 trainer/global_step‚249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 4313383155712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 4206327929792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 226665532928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 4206344969152 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4328824 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 }—r0à+ _wandbruntime‚116  _runtime‚116³©äñøáò .loss/FBt1/detection_loss‚19.95229133301311 .loss/FBt2/detection_loss‚9.953578163134424 -loss/FRt1/assignment_loss‚5.7289797202828 /loss/FRt2/assignment_loss‚5.915675509465884 1loss/SRqqt2/assignment_loss‚4.530203469445772 0loss/FBt1/assignment_loss‚2.2979517380396524 0loss/FBt2/assignment_loss‚2.1834787789144015 .loss/FRt2/detection_loss‚4.552115304625198 1loss/SRqqt2/detection_loss‚3.0989227360375575 .loss/FRt1/detection_loss‚3.802323190360954 0loss/SRqqt1/detection_loss‚3.287347005661001 epoch‚0 trainer/global_step‚249 _runtime‚ 116.168377327 _step‚19 %loss/total_loss‚69.84343612658866 2loss/SRqqt1/assignment_loss‚4.5405691776079005 % _timestamp‚1.7857934420915976e+09BÜÀüâö _step‚19 %loss/total_loss‚69.84343612658866 2loss/SRqqt1/assignment_loss‚4.5405691776079005 % _timestamp‚1.7857934420915976e+09 .loss/FBt1/detection_loss‚19.95229133301311 .loss/FBt2/detection_loss‚9.953578163134424 -loss/FRt1/assignment_loss‚5.7289797202828 /loss/FRt2/assignment_loss‚5.915675509465884 1loss/SRqqt2/assignment_loss‚4.530203469445772 0loss/FBt1/assignment_loss‚2.2979517380396524 0loss/FBt2/assignment_loss‚2.1834787789144015 .loss/FRt2/detection_loss‚4.552115304625198 1loss/SRqqt2/detection_loss‚3.0989227360375575 .loss/FRt1/detection_loss‚3.802323190360954 0loss/SRqqt1/detection_loss‚3.287347005661001 epoch‚0 trainer/global_step‚249 _runtime‚ 116.168377327ƒ½ã÷‘ã‚(:† ¥—ÄÓƦ˜¹ gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚36.0 gpu.0.powerWatts‚126.579' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚42.193 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù°UŽ[ä‚(:®  ¥—ÄÓïܺ disk.sda1.in‚0 disk.dm-0.in‚0 disk.drbd1226.out‚0" disk.nvme0n1.out‚ 4.70751953125 disk.drbd1552.out‚0 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1356.in‚0) disk./.usagePercent‚31.89558612028786 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1510.in‚0 disk.drbd1828.in‚0 disk.drbd1584.in‚0# proc.memory.rssMB‚ 11253.2265625 network.recv‚62371 disk.drbd1673.in‚0 disk.nvme0n1.in‚0 disk.drbd2078.in‚0 disk.drbd1550.out‚0 disk.drbd1938.in‚0 disk.drbd1931.in‚0 disk.drbd2118.out‚0 disk.sda.in‚0 disk.drbd1226.in‚0 disk.drbd1817.in‚0* proc.memory.availableMB‚38532.29296875 disk.drbd1938.out‚0 disk.sda2.in‚0 disk.sda2.out‚3.171875 disk.drbd1828.out‚0 proc.cpu.threads‚29 network.sent‚242388 disk.rbd1.out‚ 0.158203125 disk.rbd0.in‚0 disk.drbd1965.in‚0 disk.drbd1584.out‚0% memory_percent3HéÞ‚41.204386949539185$ disk./.usageGB‚570.3254737854004 disk.sda1.out‚0 disk.drbd1552.in‚0 disk.drbd2111.out‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 cpu‚15.785768384294498 disk.dm-0.out‚ 4.70751953125 disk.drbd1673.out‚0 disk.drbd1817.out‚0 disk.drbd1285.in‚0 disk.rbd0.out‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0* proc.memory.percent‚17.171061038970947 disk.sda.out‚3.171875X«µ.å‚(:… ´—Äӷ瓹 gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚36.0 gpu.0.powerWatts‚99.165' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚33.055 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V²÷2æ‚(:§  ´—ÄÓ’‹Ð¹ disk.sda2.out‚ 3.55859375 disk.drbd2111.in‚0 disk.drbd2118.out‚0) disk./.usagePercent‚31.89558505359866 disk.drbd1828.out‚0 proc.cpu.threads‚29 disk.drbd1828.in‚0 disk.drbd1931.out‚0 disk.dm-0.in‚0 disk.drbd1356.in‚0 disk.drbd1226.in‚0$ disk./.usageGB‚570.3254547119141 disk.drbd1356.out‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1817.in‚0 disk.rbd1.out‚ 0.158203125 network.recv‚65136 disk.drbd1552.out‚0 disk.drbd1965.in‚0 disk.rbd0.out‚0 disk.drbd1289.out‚0# proc.memory.rssMB‚ 11253.2265625# proc.memory.availableMB‚38532.5 disk.dm-0.out‚ 5.291015625 disk.drbd1938.out‚0 disk.drbd1510.in‚0 disk.drbd1285.out‚0 disk.sda2.in‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0% memory_percent‚41.204071044921875 disk.rbd0.in‚0 disk.sda.out‚ 3.55859375 disk.drbd1673.in‚0 network.sent‚287337 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.sda1.in‚0* proc.memory.percent‚17.171061038970947 disk.drbd2078.out‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 cpu‚15.627625541552074 disk.drbd1510.out‚0 disk.nvme0n1.in‚0 disk.drbd1285.in‚0 disk.sda.in‚0 disk.drbd1817.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.sda1.out‚0 disk.nvme0n1.out‚ 5.291015625 disk.drbd1289.in‚0 disk.drbd1584.in‚0 disk.drbd1226.out‚0éd´<0ç+ _wandbruntime‚138  _runtime‚138RjgÃè½ _step‚20 & lr-AdamW/pg1‚7.705883759106523e-06 & lr-AdamW/pg2‚7.705883759106523e-06 trainer/global_step‚299 % _timestamp‚1.7857934641243532e+09 _runtime‚ 138.20134489,¨¨7ÇéÁ _step‚20 & lr-AdamW/pg1‚7.705883759106523e-06 & lr-AdamW/pg2‚7.705883759106523e-06 trainer/global_step‚299 % _timestamp‚1.7857934641243532e+09 _runtime‚ 138.20134489órÖ0ê+ _wandbruntime‚138  _runtime‚138îo£À'ë‰O G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1997479 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 263660594176 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3164661 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 5167984278528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 263573164544 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 270799512064 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1997479 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5154571582464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5154554543104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5026596373472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 263013460592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1183495 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5418127707648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _runtime‚ 138.202723988 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5162138 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚860748 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 5418127707648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2044019 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1183497 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 5418232176640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3174317 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚224 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚226 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5154571582464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 5418232176640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 5438758886912 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5026613412832 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ËÒ2“'ìO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚224 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚226 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5154571582464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 5418232176640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 5438758886912 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5026613412832 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1997479 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 263660594176 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3164661 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 5167984278528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 263573164544 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 270799512064 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1997479 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5154571582464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5154554543104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5026596373472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 263013460592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1183495 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5418127707648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _runtime‚ 138.202723988 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5162138 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚860748 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 5418127707648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2044019 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1183497 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 5418232176640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3174317‰%l0í+ _wandbruntime‚138  _runtime‚138£ÀåÁ×$î“M REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚863358 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 5436278566400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5043353585152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 5436278566400 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5171721306112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2004120 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1187434 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 264452784128 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5457035959808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 264540220928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 _step‚22 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3184854 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5171738345472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 5171721306112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5179304 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2050790 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5456886680576 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5188976 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2004122 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5185191395328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5307141676482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _runtime‚ 138.574951342 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5188976 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5436174090240 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3175184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5307244519078 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5043336545792 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 264540220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2004122 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 264452784128 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚863611 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5185166491648 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1187432 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 271844564480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 263805130690 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 trainer/global_step‚299 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5171738345472 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7857934644975939e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 263890933926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2004120 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 271720188928 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 264452784128 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5457035959808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 264540220928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 _step‚22 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3184854 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5171738345472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 5171721306112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5179304 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2050790 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5456886680576 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5188976 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2004122 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5185191395328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5307141676482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _runtime‚ 138.574951342 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5188976 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5436174090240 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3175184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5307244519078 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5043336545792 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 264540220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2004122 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 264452784128 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚863611 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5185166491648 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1187432 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 271844564480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 263805130690 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 trainer/global_step‚299 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5171738345472 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7857934644975939e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 263890933926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2004120 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 271720188928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3702319 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 6320566462464 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 6012909732864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6344296672768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 307569299968 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6320461993472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 6170421110448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2329629 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1004583 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 316120520704 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚ 159.976435101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 6012892693504 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 315981457920 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2329629 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3702319 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 6320566462464 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2384930 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6344460639232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 6012909732864 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 307569299968 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 306901924100L{BêÐúŽO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 trainer/global_step‚349 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚ 159.976435101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 6012892693504 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 315981457920 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2329629 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3702319 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 6320566462464 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2384930 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6344460639232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 6012909732864 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 307569299968 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 306901924100 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2384678 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3692663 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 307656729600 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6028315214848 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2329627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5863604989472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1380345 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3702319 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 6320566462464 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 6012909732864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6344296672768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 307569299968 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6320461993472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 6170421110448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2329629 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1004583 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 316120520704 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 6338612852224 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 316883440640 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 trainer/global_step‚349 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 6030059456512 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2336272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6045522331648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 6030059456512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6362405772288 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 % _timestamp‚1.7857934862721193e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2391465 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1384282 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 6030076495872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6045547235328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6049128 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2336272 NDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 317020399104 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 6188038755990 _runtime‚ 160.349731321 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3703186 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6039456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 308448919552 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6049128 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 307779397434 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640$ Ο&ý™M QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 6030059456512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6362405772288 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 % _timestamp‚1.7857934862721193e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2391465 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1384282 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 6030076495872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6045547235328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6049128 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2336272 NDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 317020399104 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 6188038755990 _runtime‚ 160.349731321 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3703186 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6039456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 308448919552 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6049128 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 307779397434 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6039456 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3712856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 6338612852224 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 316883440640 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 trainer/global_step‚349 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 6030059456512 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2336272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6045522331648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200Þ EØ0þ+ _wandbruntime‚160  _runtime‚160øÿò trainer/global_step‚349 %loss/total_loss‚75.31876082432865 epoch‚0 /loss/FRt1/detection_loss‚3.1779826913190923 0loss/SRqqt2/detection_loss‚3.026172034856346 .loss/FBt1/detection_loss‚27.87257223862868 _step‚27 1loss/SRqqt1/assignment_loss‚4.467554043500852 1loss/SRqqt2/assignment_loss‚4.495216190814972 .loss/FRt2/detection_loss‚4.580576743930578 $ _timestamp‚1.785793486274395e+09 _runtime‚ 160.351217767 /loss/FRt1/assignment_loss‚5.704442583996316 /loss/FBt1/assignment_loss‚2.161780696648818 0loss/FBt2/assignment_loss‚2.1226554253162484 0loss/SRqqt1/detection_loss‚2.717864562558313 .loss/FBt2/detection_loss‚9.388163091471561 0loss/FRt2/assignment_loss‚5.6037805212868586zf1ü€ö %loss/total_loss‚75.31876082432865 epoch‚0 /loss/FRt1/detection_loss‚3.1779826913190923 0loss/SRqqt2/detection_loss‚3.026172034856346 .loss/FBt1/detection_loss‚27.87257223862868 _step‚27 1loss/SRqqt1/assignment_loss‚4.467554043500852 1loss/SRqqt2/assignment_loss‚4.495216190814972 .loss/FRt2/detection_loss‚4.580576743930578 $ _timestamp‚1.785793486274395e+09 _runtime‚ 160.351217767 /loss/FRt1/assignment_loss‚5.704442583996316 /loss/FBt1/assignment_loss‚2.161780696648818 0loss/FBt2/assignment_loss‚2.1226554253162484 0loss/SRqqt1/detection_loss‚2.717864562558313 .loss/FBt2/detection_loss‚9.388163091471561 0loss/FRt2/assignment_loss‚5.6037805212868586 trainer/global_step‚349¤¬œ‚(:‘ Ò—ÄÓ’õ ¹ gpu.0.gpu‚23.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.171' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.39033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒnR†7‚‚(:¬  Ò—ÄÓɤ†º disk.sda2.out‚ 4.52734375 disk.drbd2118.out‚0* disk./.usagePercent‚31.895725003222253 disk.drbd1510.in‚0 disk.rbd0.in‚0 disk.drbd1673.in‚0 disk.drbd1817.out‚0( proc.memory.availableMB‚ 38531.953125* proc.memory.percent‚17.171061038970947 proc.cpu.threads‚29 disk.rbd1.in‚0 disk.sda2.in‚0 disk.drbd2078.in‚0 disk.drbd1550.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1931.in‚0 disk.nvme0n1.in‚0 disk.drbd1285.out‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1289.in‚0 disk.drbd1226.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1817.in‚0 cpu‚15.429256884984463 disk.nvme0n1.out‚ 6.787109375 disk.drbd1285.in‚0 disk.drbd2111.out‚0 disk.drbd1938.in‚0 network.recv‚68935 disk.drbd1965.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.rbd1.out‚ 0.158203125 disk.drbd1584.in‚0 disk.sda.in‚0 disk.dm-0.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1356.in‚0 disk.sda.out‚ 4.52734375 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.rbd0.out‚0 disk.drbd2078.out‚0 disk.drbd1550.in‚0 disk.drbd1828.out‚0 disk.dm-0.out‚ 6.787109375 disk.drbd1938.out‚0$ disk./.usageGB‚570.3279571533203 disk.drbd2111.in‚0 disk.sda1.out‚0 disk.drbd1931.out‚0$ memory_percent‚41.20490550994873# proc.memory.rssMB‚ 11253.2265625 disk.drbd1226.out‚0 network.sent‚335686 disk.drbd1552.out‚0+WK›ƒ‚(: á—ÄÓ—ó”¹ gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚37.0 gpu.0.powerWatts‚94.976' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚31.65866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù/ÑY1„‚(:¦  á—ÄÓè–ã¹ disk.drbd1828.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1226.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0* proc.memory.percent‚17.171061038970947 disk.nvme0n1.in‚0! disk.nvme0n1.out‚ 7.8857421875 disk.sda2.in‚0 disk.dm-0.in‚0 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.drbd1931.out‚0 cpu‚15.343920338471916 proc.cpu.threads‚29 network.recv‚71377 disk.drbd1552.out‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.drbd1584.out‚0 disk.sda.out‚4.9375# proc.memory.rssMB‚ 11253.2265625 network.sent‚380657) disk./.usagePercent‚31.89572073646544 disk.drbd1510.out‚0 disk.drbd1356.out‚0 disk.drbd2078.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1289.out‚0 disk.drbd1584.in‚0 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.rbd1.out‚ 0.158203125 disk.drbd2118.in‚0 disk.drbd2041.out‚0# disk./.usageGB‚570.327880859375 disk.drbd1510.in‚0 disk.sda.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0* proc.memory.availableMB‚38531.85546875 disk.drbd1285.out‚0 disk.drbd1965.out‚0 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd2041.in‚0 disk.sda1.in‚0 disk.dm-0.out‚ 7.8857421875$ memory_percent‚41.20505452156067 disk.drbd1931.in‚0 disk.drbd1673.out‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.sda2.out‚4.9375Ùñ»0…+ _wandbruntime‚183  _runtime‚183ÿ³Ô ņ¿ ' lr-AdamW/pg2‚1.0283102407637132e-05 trainer/global_step‚399 % _timestamp‚1.7857935099007423e+09 _runtime‚ 183.97771215 _step‚28 ' lr-AdamW/pg1‚1.0283102407637132e-05S»›7ɇà _step‚28 ' lr-AdamW/pg1‚1.0283102407637132e-05 ' lr-AdamW/pg2‚1.0283102407637132e-05 trainer/global_step‚399 % _timestamp‚1.7857935099007423e+09 _runtime‚ 183.97771215Ãh7Q0ˆ+ _wandbruntime‚183  _runtime‚183 V»¦±‰‹O QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7222900748288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7222900748288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2661777 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6892100 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2725603 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 351565435392 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 361161851392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚29 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7222796279296 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 351652865024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 6871230843904 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7051318189956 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 6700613605472 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 L=DeviceStatsMonitor.on_train_bat€†€àch_start/active_bytes.all.peak‚ 2940455936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2661777 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4230321 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2661779 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7051421032440 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 6871230843904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816½ÝÓw•'ŠO \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 7249990581248 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 350790387608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4220665 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 351565435392 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 350704584484 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4230321 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2725860 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4220665 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7222900748288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7222900748288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2661777 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6892100 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2725603 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 351565435392 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 361161851392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚29 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7222796279296 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 351652865024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 6871230843904 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7051318189956 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 6700613605472 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2661777 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4230321 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2661779 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7051421032440 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 6871230843904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 H0‹+ _wandbruntime‚184  _runtime‚184Ÿ©© &ŒšM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2668420 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6909280 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 6717353777792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2668420 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4240858 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 362195410432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 6888414646272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1581134 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4231188 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 362062646272 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6899608 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7068935835498 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1581132 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7069038678094 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2732382 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 352532491776 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1151250 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6905878171648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _step‚30 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 352445054976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2732643 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 351667860942 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 6888414646272 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7240947138048 % _timestamp‚1.7857935102812092e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7267940817920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6905903075328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6899608 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2668422 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 351582057706 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251ÿk蘞M PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1581132 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7069038678094 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2732382 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 352532491776 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1151250 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6905878171648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _step‚30 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 352445054976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2732643 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 351667860942 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 6888414646272 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7240947138048 % _timestamp‚1.7857935102812092e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7267940817920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6905903075328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6899608 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2668422 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 351582057706 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2668420 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6909280 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 6717353777792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2668420 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocat8þñ ed‚4240858 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 362195410432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 6888414646272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1581134 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4231188 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 362062646272 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6899608 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7068935835498 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 27682406404K )0Ž+ _wandbruntime‚184  _runtime‚184€ÓK©øò 1loss/SRqqt1/assignment_loss‚4.612997987895336 1loss/SRqqt2/assignment_loss‚4.568404649579247 .loss/FRt2/detection_loss‚5.528022283901934 % _timestamp‚1.7857935102831612e+09 0loss/FBt2/assignment_loss‚2.0085383809131123 .loss/FRt1/detection_loss‚3.425787265197589 _runtime‚ 184.359955411 _step‚31 0loss/SRqqt1/detection_loss‚3.128927069526274 0loss/SRqqt2/detection_loss‚3.896704543468564 /loss/FBt2/detection_loss‚16.555954834041387 trainer/global_step‚399 /loss/FRt2/assignment_loss‚5.649252227095307 0loss/FBt1/assignment_loss‚2.1657698154449463 /loss/FBt1/detection_loss‚120.27723624308904 $loss/total_loss‚177.611319250172 epoch‚0 .loss/FRt1/assignment_loss‚5.79372395001925DX»ïüö 0loss/FBt2/assignment_loss‚2.0085383809131123 .loss/FRt1/detection_loss‚3.425787265197589 _runtime‚ 184.359955411 _step‚31 0loss/SRqqt1/detection_loss‚3.128927069526274 0loss/SRqqt2/detection_loss‚3.896704543468564 /loss/FBt2/detection_loss‚16.555954834041387 trainer/global_step‚399 /loss/FRt2/assignment_loss‚5.649252227095307 0loss/FBt1/assignment_loss‚2.1657698154449463 /loss/FBt1/detection_loss‚120.27723624308904 $loss/total_loss‚177.611319250172 epoch‚0 .loss/FRt1/assignment_loss‚5.79372395001925 1loss/SRqqt1/assignment_loss‚4.612997987895336 1loss/SRqqt2/assignment_loss‚4.568404649579247 .loss/FRt2/detection_loss‚5.528022283901934 % _timestamp‚1.7857935102831612e+093„Û‘‘‚(:† ð—ÄÓ¨÷ñ¸ gpu.0.gpu‚10.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.848' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚41.616 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áyf8’‚(:­  ð—ÄÓø¥µ¹ disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1828.out‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.rbd1.out‚ 0.158203125 disk.sda.in‚0 disk.sda1.in‚0 proc.cpu.threads‚29 disk.drbd1510.in‚0 disk.drbd1285.in‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0* disk./.usagePercent‚31.895717323059987$ disk./.usageGB‚570.3278198242188! disk.nvme0n1.out‚ 8.1396484375 disk.drbd1552.in‚0 disk.rbd1.in‚0 disk.sda2.out‚ 5.47265625 disk.drbd1938.out‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0( proc.memory.availableMB‚ 38531.265625# proc.memory.rssMB‚ 11253.2265625 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 5.47265625 disk.drbd2041.in‚0 disk.dm-0.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚0 network.sent‚383941 network.recv‚72614 disk.drbd1817.in‚0 disk.rbd0.out‚0 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚0 disk.drbd1285.out‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 cpu‚15.24210679224781 disk.drbd1289.in‚0 disk.drbd1931.out‚0* proc.memory.percent‚17.171061038970947 disk.drbd1226.out‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1673.in‚0 disk.drbd2118.in‚0 disk.drbd1226.in‚0 disk.drbd2111.in‚0 disk.dm-0.out‚ 8.1396484375 disk.drbd1550.out‚0$ memory_percent‚41.20595455169678æ’ T0“+ _wandbruntime‚206  _runtime‚206û°êÆ”À trainer/global_step‚449 % _timestamp‚1.7857935319392443e+09 _runtime‚ 206.016194499 _step‚32 ' lr-AdamW/pg1‚1.1571711731902438e-05 ' lr-AdamW/pg2‚1.1571711731902438e-05FÊ•Ä _runtime‚ 206.016194499 _step‚32 ' lr-AdamW/pg1‚1.1571711731902438e-05 ' lr-AdamW/pg2‚1.1571711731902438e-05 trainer/global_step‚449 % _timestamp‚1.7857935319392443e+09 V²åC0–+ _wandbruntime‚206  _runtime‚206ÌAßV‘'—‹O JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8125130565120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 7749051798528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚ 206.017519743 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2993929 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8125235034112 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚33 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 395561570816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2993927 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8125130565120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7729586033664 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4748667 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3065401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 406493060608 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7932215269464 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 395561570816 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7752252 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2993929 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 8155544859136 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8155389281280 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2993927 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1774045 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1291609 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7742594 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 406362386432 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7857935319402637e+09 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7932318111948 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7729568994304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8125130565120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7729586033664 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4748667 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3065401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 406493060608 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7932215269464 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 395561570816 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7752252 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2993929 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 8155544859136 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8155389281280 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2993927 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1774045 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1291609 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7742594 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 406362386432 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7857935319402637e+09 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7932318111948 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7729568994304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 H8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7746752796672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7769432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4759190 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7759760 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7554362393792 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4768860 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8143281423872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 396528627200 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 396441190400 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1777984 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚34 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1777982 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3000572 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3072184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3000570 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 396528627200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7769432 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3072435 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8143176947712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1294451 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 395556324450 _runtime‚ 206.386421484 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚'k<0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 trainer/global_step‚449 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7949935757602 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 396441190400 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8143281423872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4768860 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 407398919168 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3000570 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 N8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7746752796672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7769432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4759190 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7759760 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7554362393792 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4768860 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8143281423872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 396528627200 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 396441190400 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1777984 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚34 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1777982 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3000572 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3072184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3000570 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 396528627200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7769432 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3072435 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8143176947712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1294451 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 395556324450 _runtime‚ 206.386421484 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 trainer/global_step‚449 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7949935757602 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 396441190400 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8143281423872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4768860 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 407398919168 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3000570 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 9027464850944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8602746 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8609382734848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 439557706240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1433856 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3326077 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9060899269632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 8587924184064 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3404751 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 9027464850944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8609407638528 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 8813112348972 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 % _timestamp‚1.7857935540181231e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 8587924184064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 439557706240 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9061063236096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 8374630837472 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1970895 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5286325 trainer/global_step‚499 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 8587907144704 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 438481511500 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3326079 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5286325 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 438567314624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 9027569319936 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1970897 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 451516534784 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8612404 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 8813215191456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0DDþçw ¨O SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 9027464850944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8609407638528 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 8813112348972 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 % _timestamp‚1.7857935540181231e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 8587924184064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 439557706240 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9061063236096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 8374630837472 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1970895 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5286325 trainer/global_step‚499 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 8587907144704 LDDeviceStatsMonitor.on_train_b¿úqVatch_start/segment.large_pool.allocated‚130 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 438481511500 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3326079 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5286325 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 438567314624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 9027569319936 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1970897 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 451516534784 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8612404 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 8813215191456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 9027464850944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8602746 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8609382734848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 439557706240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1433856 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3326077 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9060899269632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 8587924184064 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3404751%&‚“0©+ _wandbruntime‚228  _runtime‚228  ›&ª•M C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _step‚38 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 452553296896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 440524762624 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8619912 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1974832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 8605090947072 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 440437325824 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1974834 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 439444787958 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9079008287232 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 9045511233536 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3332720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 8391371009792 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5287192 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8629584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 % _timestamp‚1.7857935543961484e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3332720 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1436720 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8626614755328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8619912 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5296862 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 9045615709696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _runtime‚ 228.473532583 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 8605090947072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5287192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8629584 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3332722 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3332722 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9079168052224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8626589851648 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 9045615709696 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 trainer/global_step‚499 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 440524762624 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5296862 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1436962 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 440437325824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3411796 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3411552 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 439358984722 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5296862 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 9045615709696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _runtime‚ 228.473532583 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 8605090947072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5287192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8629584 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3332722 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3332722 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9079168052224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8626589851648 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 9045615709696 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 trainer/global_step‚499 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 440524762624 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5296862 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1436962 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 440437325824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3411796 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3411552 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 439358984722 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8629584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 % _timestamp‚1.7857935543961484e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3332720 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1436720 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8626614755328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130&–'n„0¬+ _wandbruntime‚228  _runtime‚228þqZù­ó %loss/total_loss‚99.21463469266025 /loss/FRt1/assignment_loss‚5.904673673773325 /loss/FRt2/assignment_loss‚5.519016499789256 1loss/SRqqt1/assignment_loss‚4.498192935078232 0loss/FBt1/assignment_loss‚1.9202178716659546 0loss/FBt2/assignment_loss‚1.5805921670867176 /loss/FRt1/detection_loss‚3.8933441687014794 .loss/FRt2/detection_loss‚6.288226018536766 0loss/SRqqt2/detection_loss‚3.009863005045357 .loss/FBt1/detection_loss‚50.53274565935135 trainer/global_step‚499 _runtime‚ 228.475133804 _step‚39 .loss/FBt2/detection_loss‚8.831817552719901 epoch‚0 1loss/SRqqt1/detection_loss‚2.7940178610108517 1loss/SRqqt2/assignment_loss‚4.441927279901067 $ _timestamp‚1.785793554398338e+09ý?œý®÷ %loss/total_loss‚99.21463469266025 /loss/FRt1/assignment_loss‚5.904673673773325 /loss/FRt2/assignment_loss‚5.519016499789256 1loss/SRqqt1/assignment_loss‚4.498192935078232 0loss/FBt1/assignment_loss‚1.9202178716659546 0loss/FBt2/assignment_loss‚1.5805921670867176 /loss/FRt1/detection_loss‚3.8933441687014794 .loss/FRt2/detection_loss‚6.288226018536766 0loss/SRqqt2/detection_loss‚3.009863005045357 .loss/FBt1/detection_loss‚50.53274565935135 trainer/global_step‚499 _runtime‚ 228.475133804 _step‚39 .loss/FBt2/detection_loss‚8.831817552719901 epoch‚0 1loss/SRqqt1/detection_loss‚2.7940178610108517 1loss/SRqqt2/assignment_loss‚4.441927279901067 $ _timestamp‚1.785793554398338e+09'ç¼7¯‚(:¬  ˜ÄÓ߆޹ disk.drbd1289.out‚0 disk.drbd1584.out‚0 disk.sda.out‚6.703125 disk.drbd1931.out‚0 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0* disk./.usagePercent‚31.895689802478515 disk.drbd1356.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1673.in‚0) proc.memory.availableMB‚ 38530.0859375 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd2041.out‚0! disk.nvme0n1.out‚ 9.3896484375* proc.memory.percent‚17.171061038970947 disk.drbd1285.in‚0 disk.drbd1356.out‚0 disk.sda2.out‚6.703125 disk.drbd1226.out‚0 disk.drbd1817.out‚0 proc.cpu.threads‚29 network.sent‚477442$ disk./.usageGB‚570.3273277282715 disk.drbd1510.out‚0 disk.rbd0.in‚0 disk.drbd1584.in‚0 disk.rbd0.out‚0 disk.drbd1510.in‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1938.out‚0 disk.drbd1817.in‚0# proc.memory.rssMB‚ 11253.2265625 disk.sda2.in‚0 disk.drbd1289.in‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1285.out‚0 disk.sda.in‚0 disk.dm-0.in‚0% memory_percent‚41.207754611968994 cpu‚14.994229623622159 network.recv‚79153 disk.rbd1.out‚ 0.158203125 disk.drbd2118.in‚0 disk.sda1.in‚0 disk.drbd1226.in‚0 disk.nvme0n1.in‚0 disk.drbd2078.out‚0 disk.dm-0.out‚ 9.3896484375&ÀwŒœ°‚(:‘ ˜ÄÓÌÍö¹ gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚37.0 gpu.0.powerWatts‚123.649' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.21633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z\òæ0±+ _wandbruntime‚250  _runtime‚250ÎY.1ƲÀ trainer/global_step‚549 % _timestamp‚1.7857935762484043e+09 _runtime‚ 250.325360701 _step‚40 ' lr-AdamW/pg1‚1.4148930380433047e-05 ' lr-AdamW/pg2‚1.4148930380433047e-05¡ƒÊ³Ä _runtime‚ 250.325360701 _step‚40 ' lr-AdamW/pg1‚1.4148930380433047e-05 ' lr-AdamW/pg2‚1.4148930380433047e-05 trainer/global_step‚549 % _timestamp‚1.7857935762484043e+09(Ê|ñ0´+ _wandbruntime‚250  _runtime‚250Û¿²-µ‹O C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2167745 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 9469738574848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9211639453472 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 9929799136768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ^MÈmãûd#DeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 496742432256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 9929903605760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 9446262334464 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5804671 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2167747 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 482369975008 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 9469763478528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9211656492832 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3658229 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9966481007104 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 9694009428480 trainer/global_step‚549 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 9929903605760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1577298 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 496873106432 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5814327 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 483641271296 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3745308 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9472556 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1577561 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 9446262334464 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 9929799136768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 483553841664 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 _runtime‚ 250.326739907 _step‚41 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3658227 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 % _timestamp‚1.7857935762494674e+09 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5814327 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5804671 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 483641271296 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 9446245295104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 482455778132 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121Õ¥I•'¶O P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 9929799136768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 496742432256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 9929903605760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 9446262334464 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5804671 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2167747 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 482369975008 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 9469763478528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9211656492832 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3658229 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9966481007104 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 9694009428480 trainer/global_step‚549 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 9929903605760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1577298 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 496873106432 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5814327 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 483641271296 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3745308 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9472556 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1577561 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 9446262334464 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 9929799136768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 483553841664 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 _runtime‚ 250.326739907 _step‚41 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3658227 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 % _timestamp‚1.7857935762494674e+09 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5814327 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5804671 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 483641271296 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 9446245295104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 482455778132 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2167745 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 9469738574848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9211639453472)ea•J0·+ _wandbruntime‚250  _runtime‚250ÈC~›&¸•M SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9489736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9489736 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9984748249600 _step‚42 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3664870 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5824864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 9947949995520 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 9228396665152 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 483247448230 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1580133 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 497646987264 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9480064 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 497777654272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9480064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 9463429097472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9463412058112 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3751815 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9984592678912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5815194 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 9463429097472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 484520898048 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3664872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1580385 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 9463412058112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9486970595328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 9486945691648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856µ´¹™M B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9480064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 9463429097472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9463412058112 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3751815 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9984592678912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5815194 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 9463429097472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 484520898048 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3664872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1580385 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 9463412058112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9486970595328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 9486945691648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9489736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9489736 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9984748249600 _step‚42 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3664870 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5824864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 9947949995520 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 9228396665152 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 483247448230 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1580133 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 497646987264 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9480064 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 497777654272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251*S 0º+ _wandbruntime‚250  _runtime‚250OÓ{Íö»ð _step‚43 1loss/SRqqt1/assignment_loss‚4.422692975093579 1loss/SRqqt2/detection_loss‚3.5230563593671675 &loss/total_loss‚137.60918369797793 epoch‚0 trainer/global_step‚549 _runtime‚ 250.715218217 1loss/SRqqt2/assignment_loss‚4.389588292608869 0loss/FBt1/assignment_loss‚1.6120378971099854 .loss/FRt1/detection_loss‚3.778728397873541 .loss/FRt2/detection_loss‚5.076332896030866 /loss/SRqqt1/detection_loss‚2.49172627913027 .loss/FBt2/detection_loss‚11.29584879721656 /loss/FRt1/assignment_loss‚5.834993387262027 /loss/FRt2/assignment_loss‚5.508192392495962 /loss/FBt2/assignment_loss‚1.228833252733404 -loss/FBt1/detection_loss‚88.4471527710557 $ _timestamp‚1.785793576638313e+09*¥‘ú¼ô 0loss/FBt1/assignment_loss‚1.6120378971099854 .loss/FRt1/detection_loss‚3.778728397873541 .loss/FRt2/detection_loss‚5.076332896030866 /loss/SRqqt1/detection_loss‚2.49172627913027 .loss/FBt2/detection_loss‚11.29584879721656 /loss/FRt1/assignment_loss‚5.834993387262027 /loss/FRt2/assignment_loss‚5.508192392495962 /loss/FBt2/assignment_loss‚1.228833252733404 -loss/FBt1/detection_loss‚88.4471527710557 $ _timestamp‚1.785793576638313e+09 _step‚43 1loss/SRqqt1/assignment_loss‚4.422692975093579 1loss/SRqqt2/detection_loss‚3.5230563593671675 &loss/total_loss‚137.60918369797793 epoch‚0 trainer/global_step‚549 _runtime‚ 250.715218217 1loss/SRqqt2/assignment_loss‚4.389588292608869+ÏGšyœ½‚(:‘ ¬˜ÄÓë§ý¸ gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚37.0 gpu.0.powerWatts‚121.613' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.53766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OãÎ6;¾‚(:°  ¬˜ÄÓô¿¹ disk.drbd1226.out‚0 disk.drbd1817.out‚0 disk.sda2.in‚0 disk.sda2.out‚ 7.17578125 disk.drbd2078.in‚0 disk.drbd1550.in‚0 disk.drbd1938.in‚0# proc.memory.rssMB‚ 11253.2265625 cpu‚14.92847234698351 disk.drbd1552.out‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 network.sent‚522617 disk.drbd1673.in‚0 disk.drbd1226.in‚0* disk./.usagePercent‚31.895717109722142 disk.nvme0n1.in‚0 disk.rbd0.out‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.dm-0.in‚0 disk.dm-0.out‚ 9.5419921875 disk.drbd2111.in‚0 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd1510.in‚0* proc.memory.availableMB‚38530.30859375 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1965.out‚0 disk.drbd1584.out‚0 proc.cpu.threads‚29$ disk./.usageGB‚570.3278160095215 disk.sda.out‚ 7.17578125 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd1817.in‚0 disk.drbd1510.out‚0* proc.memory.percent‚17.171061038970947 network.recv‚82192 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1931.out‚0! disk.nvme0n1.out‚ 9.5419921875 disk.rbd0.in‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0 disk.drbd1552.in‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd1938.out‚0 disk.drbd1673.out‚0% memory_percent‚41.207414865493774 disk.rbd1.out‚ 0.158203125 disk.drbd1550.out‚0 disk.drbd2118.in‚0éõF«¿‚(:’ »˜ÄÓÔÀ™¹ gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚37.0 gpu.0.powerWatts‚100.213' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚33.404333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z´l/À‚(:¤  »˜ÄÓ›óâ¹! disk.nvme0n1.out‚ 10.353515625 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.sda.in‚0* proc.memory.percent‚17.171061038970947$ disk./.usageGB‚570.3275909423828 disk.drbd2111.in‚0$ memory_percent‚41.20742082595825 disk.drbd1552.in‚0 disk.rbd1.out‚ 0.158203125 disk.drbd1289.in‚0 disk.drbd1226.in‚0 disk.nvme0n1.in‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1510.in‚0 disk.rbd0.in‚0 disk.drbd1828.in‚0 disk.dm-0.out‚ 10.353515625 disk.drbd1510.out‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 network.recv‚84865 disk.drbd1356.out‚0 disk.sda2.in‚0 disk.sda2.out‚8.125 disk.drbd1817.out‚0 disk.dm-0.in‚0 proc.cpu.threads‚29 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1931.in‚0) proc.memory.availableMB‚ 38530.3046875 cpu‚14.867411445639142) disk./.usagePercent‚31.89570452278953 disk.drbd1938.in‚0 disk.drbd1673.in‚0 disk.drbd1828.out‚0 disk.rbd0.out‚0 disk.drbd1584.out‚0 disk.drbd2041.in‚0# proc.memory.rssMB‚ 11253.2265625 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.sda.out‚8.125 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd1226.out‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd1356.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 network.sent‚567569 disk.drbd1938.out‚0ÂpHc0Á+ _wandbruntime‚272  _runtime‚272AûáËÆÂÀ ' lr-AdamW/pg1‚1.5437539704698353e-05 ' lr-AdamW/pg2‚1.5437539704698353e-05 trainer/global_step‚599 % _timestamp‚1.7857935982729218e+09 _runtime‚ 272.349873121 _step‚44ѯ“úÊÃÄ ' lr-AdamW/pg1‚1.5437539704698353e-05 ' lr-AdamW/pg2‚1.5437539704698353e-05 trainer/global_step‚599 % _timestamp‚1.7857935982729218e+09 _runtime‚ 272.349873121 _step‚44,rP¤t0Ä+ _wandbruntime‚272  _runtime‚272Þ¢tv¥'ÅŸO TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚10574906507988 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 542055262208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6342329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 527637406720 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6332673 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚10304583445504 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 trainer/global_step‚599 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1721048 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10872021100032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 527549977088 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚10304600484864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10330094414848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚10048648069472 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 526258438516 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚10832237891584 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2364597 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6332673 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3990377 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10832133422592 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10872174580736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚10304583445504 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 527549977088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6342329 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4085643 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚10575009350472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _runtime‚ 272.351242246 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 527637406720 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3990379 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 541926685184 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚10332708 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4085894 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 526344241640 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚10048648069472 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 526258438516 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚10832237891584 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2364597 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6332673 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3990377 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10832133422592 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10872174580736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚10304583445504 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 527549977088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6342329 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4085643 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚10575009350472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _runtime‚ 272.351242246 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 527637406720 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3990379 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 541926685184 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚10332708 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4085894 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 526344241640 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 527637406720 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6332673 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚10304583445504 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 trainer/global_step‚599 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1721048 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10872021100032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 527549977088 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚10304600484864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10330094414848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251-& ÞÇ+ _wandb½sª runtime‚272  _runtime‚272ÛO¹å¯&È©M YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚10321750208512 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚10592626996126 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚10592524153530 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6352866 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3997022 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1723877 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3997022 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚10321767247872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10890128307200 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 542826775552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10347326435328 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 542961636864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚10321750208512 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3997020 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2368534 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 528429596672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10349888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚46 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 528517033472 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3997020 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚10065388241792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚ 272.715921661 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4092409 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚10065405281152 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 527221714974 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 528429596672 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10349888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1724129 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10340216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚10321767247872 7DO³&É­M ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 542961636864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚10321750208512 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3997020 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2368534 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 528429596672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10349888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚46 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 528517033472 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3997020 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚10065388241792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚ 272.715921661 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4092409 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚10065405281152 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 527221714974 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 528429596672 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10349888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1724129 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10340216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚10321767247872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚10321750208512 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚10592626996126 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚10592524153530 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6352866 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3997022 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1723877 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3997022 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚10321767247872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10890128307200 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 542826775552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10347326435328 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251.Ç:é%0Ê+ _wandbruntime‚272  _runtime‚272=ÝÖúËô 0loss/SRqqt1/detection_loss‚2.983954349277066 .loss/FBt2/detection_loss‚8.285237129589188 trainer/global_step‚599 1loss/SRqqt1/assignment_loss‚4.109219726394205 %loss/total_loss‚66.47763124801084 epoch‚0 _runtime‚ 272.716570067 _step‚47 1loss/SRqqt2/assignment_loss‚4.173827142131572 0loss/FBt1/assignment_loss‚1.2332783937454224 0loss/FBt2/assignment_loss‚0.9469784597555796 /loss/FRt1/detection_loss‚3.0705746029291237 .loss/FRt2/detection_loss‚5.197411847019953 1loss/SRqqt2/detection_loss‚3.2746028742011712 .loss/FBt1/detection_loss‚21.77539057098329 % _timestamp‚1.7857935986397765e+09 /loss/FRt1/assignment_loss‚5.886844066151402 /loss/FRt2/assignment_loss‚5.540312085832868Ñ%³þÌø epoch‚0 _runtime‚ 272.716570067 _step‚47 1loss/SRqqt2/assignment_loss‚4.173827142131572 0loss/FBt1/assignment_loss‚1.2332783937454224 0loss/FBt2/assignment_loss‚0.9469784597555796 /loss/FRt1/detection_loss‚3.0705746029291237 .loss/FRt2/detection_loss‚5.197411847019953 1loss/SRqqt2/detection_loss‚3.2746028742011712 .loss/FBt1/detection_loss‚21.77539057098329 % _timestamp‚1.7857935986397765e+09 /loss/FRt1/assignment_loss‚5.886844066151402 /loss/FRt2/assignment_loss‚5.540312085832868 0loss/SRqqt1/detection_loss‚2.983954349277066 .loss/FBt2/detection_loss‚8.285237129589188 trainer/global_step‚599 1loss/SRqqt1/assignment_loss‚4.109219726394205 %loss/total_loss‚66.47763124801084/Õ-÷›Í‚(: ʘÄÓÊÞñ¸ gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚120.44' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.14666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…‡õ7΂(:¬  ʘÄÓ…‘º¹ disk.drbd1938.out‚0 disk.drbd1817.out‚0 disk.sda2.in‚0 disk.sda2.out‚ 8.50390625 disk.drbd1289.in‚0 cpu‚14.811256128041958 disk.drbd1817.in‚0 network.sent‚570744 disk.drbd1226.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚85996 disk.drbd2118.in‚0! disk.nvme0n1.out‚ 10.568359375 disk.drbd1289.out‚0* proc.memory.percent‚17.171061038970947 proc.cpu.threads‚29 disk.drbd1938.in‚0 disk.drbd1550.out‚0$ memory_percent‚41.20798110961914 disk.drbd1584.out‚0$ disk./.usageGB‚570.3276138305664 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.rbd1.out‚ 0.158203125 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.dm-0.out‚ 10.568359375 disk.drbd1226.in‚0 disk.drbd1356.out‚0 disk.drbd1552.out‚0 disk.drbd2078.in‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.drbd2041.out‚0 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1550.in‚0 disk.sda.out‚ 8.50390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚0 disk.drbd1552.in‚0 disk.drbd2111.in‚0# proc.memory.rssMB‚ 11253.2265625 disk.sda.in‚0 disk.drbd1285.in‚0& proc.memory.availableMB‚ 38529.9375 disk.drbd1828.out‚0 disk.nvme0n1.in‚0 disk.drbd1285.out‚0 disk.drbd1828.in‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0* disk./.usagePercent‚31.895705802816575 disk.drbd1965.in‚0¸ÒÁÁ0Ï+ _wandbruntime‚294  _runtime‚294–ý6wÅп ' lr-AdamW/pg1‚1.6726149028963658e-05 ' lr-AdamW/pg2‚1.6726149028963658e-05 trainer/global_step‚649 $ _timestamp‚1.785793620241003e+09 _runtime‚ 294.317961016 _step‚48ìÏ=ªÉÑà ' lr-AdamW/pg1‚1.6726149028963658e-05 ' lr-AdamW/pg2‚1.6726149028963658e-05 trainer/global_step‚649 $ _timestamp‚1.785793620241003e+09 _runtime‚ 294.317961016 _step‚480· " 0Ò+ _wandbruntime‚294  _runtime‚294ž™\;"ÓŸO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11455803587496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 571633542144 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚11162938635264 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11455906429980 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 570146902024 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6860675 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚11777510671872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11190450254848 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 % _timestamp‚1.7857936202420192e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 587193188352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _runtime‚ 294.322647032 _step‚49 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4322529 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1865366 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚10885673724832 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11734467708416 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6870331 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6870331 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11734572177408 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 571546112512 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11162938635264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2561447 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4322527 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11190475158528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDevicežßZÕjStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 571546112512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680ì Ó©'Ô£O ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 587060417024 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 trainer/global_step‚649 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11734467708416 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6870331 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6870331 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11734572177408 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 571546112512 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11162938635264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2561447 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4322527 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11190475158528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 571546112512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11455803587496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 571633542144 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚11162938635264 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11455906429980 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 570146902024 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6860675 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚11777510671872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11190450254848 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 % _timestamp‚1.7857936202420192e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 587193188352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _runtime‚ 294.322647032 _step‚49 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4322529 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1865366 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚10885673724832 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11200368 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6871198 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚11752618567168 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6880868 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 572513168896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 _step‚50 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2565384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6871198 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚11752618567168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 571110178482 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2565382 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4329172 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11473421233038 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4433296 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1868154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚11473524075634 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚10902396857792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11180088358912 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 572425732096 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11210040 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11180088358912 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚10902413897152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200vj½³&×­M XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 572513168896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 587968756736 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2565384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6871198 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚11752618567168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 571110178482 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2565382 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4329172 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11473421233038 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4433296 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1868154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚11473524075634 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚10902396857792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11180088358912 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 572425732096 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11210040 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11180088358912 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚10902413897152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11200368 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6871198 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚11752618567168 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6880868 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 572513168896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 _step‚50 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4765568 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12636801994240 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚12021259746304 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 615629677568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 615629677568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2007271 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12683207156736 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 614121168656 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚12021259746304 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11722682340832 _runtime‚ 316.293166688 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7398333 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7388677 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 614035365532 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2758295 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12636906463232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4765325 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚12021276785664 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2758297 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4654677 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12683043190272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12050830998528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚699 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392ý‘ã¨'ä¢O SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2758295 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12636906463232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4765325 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚12021276785664 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2758297 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4654677 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12683043190272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12050830998528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚699 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4654679 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚12636801994240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚12336700667004 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 615542247936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚12336803509488 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _step‚53 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12043354 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚12636906463232 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12053012 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4765568 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12636801994240 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚12021259746304 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 615629677568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 615629677568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2007271 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12683207156736 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 614121168656 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚12021259746304 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11722682340832 _runtime‚ 316.293166688 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7398333 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7388677 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 614035365532 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚05<òý0å+ _wandbruntime‚316  _runtime‚316®Ã¦‘æ¨M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12070192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12654848376832 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7408870 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12701309826560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12060520 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11739422513152 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 614912838754 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚12038426509312 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 614998641990 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 % _timestamp‚1.7857936425907152e+09 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7399200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4772012 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4661322 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 trainer/global_step‚699 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 616421867520 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚12654952852992 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12060520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11739405473792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2762232 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚12038443548672 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 616509304320 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 _runtime‚ 316.66808767 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚12654952852992 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 616421867520 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2762234 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 616509304320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 633145238528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/ovi8¬Æ ersize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚54 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12068038115328 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚12354421155142 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12070192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚12038443548672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4661320 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2009780 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12070192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12654848376832 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7408870 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12701309826560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12060520 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11739422513152 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 614912838754 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚12038426509312 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 614998641990 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 % _timestamp‚1.7857936425907152e+09 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7399200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4772012 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4661322 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 trainer/global_step‚699 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 616421867520 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚12654952852992 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12060520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11739405473792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2762232 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚12038443548672 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 616509304320 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 _runtime‚ 316.66808767 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚12654952852992 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 616421867520 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2762234 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 616509304320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 633145238528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚54 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12068038115328 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚12354421155142 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12070192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚12038443548672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4661320 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2009780 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12911186838528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚12559673917472 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5105843 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2955147 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12913164 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2955145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚13539136280064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚13539240749056 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 658009632164 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7916679 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7916679 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 677582898688 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4986827 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚12879614936064 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4986827 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7926335 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12911161934848 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 659625812992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12911186838528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚12559673917472 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 H>DeviceStatsMonitor.onp‘rwW_train_batch_start/active.small_pool.peak‚11816 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5105843 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2955147 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12913164 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2955145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚13539136280064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚13539240749056 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 658009632164 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7916679 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7916679 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 677582898688 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4986827 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚12879614936064 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4986827 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7926335 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12911161934848 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 2537553929gh[0ó+ _wandbruntime‚339  _runtime‚339%Uìl¯&ô©M E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 678483155968 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚13557287138816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13606727832064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚12896764659712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚12896764659712 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5112896 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13606877111296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 trainer/global_step‚749 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2153812 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2153571 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚13557182662656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚12896781699072 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12930344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7927202 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2959084 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 678358780416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚12576414089792 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12920672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚12896781699072 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4993470 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2959082 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5112653 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7936872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4993470 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0>ÿ,K³&õ­M ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚12576431129152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7936872 _runtime‚ 339.050209148 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4993472 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5112653 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7936872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4993470 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 678483155968 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚13557287138816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13606727832064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚12896764659712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚12896764659712 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5112896 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13606877111296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 trainer/global_step‚749 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2153812 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2153571 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚13557182662656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚12896781699072 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12930344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7927202 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2959084 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 678358780416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚12576414089792 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12920672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚12896781699072 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4993470 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2959082 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8454337 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14441470565888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13737953086464 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3151995 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13396682533472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train· ²¼_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5318977 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8444681 % _timestamp‚1.7857936862430165e+09 _runtime‚ 360.320170646 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14441470565888 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚14494192139264 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14441575034880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5318979 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13737936047104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13771517774848 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 703621948416 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5446243 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚14494351911424 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚14098494826020 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 701812292548 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚799 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚13737953086464 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13771542678528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 703534518784 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13773316 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 722674364416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5318977 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5446512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5318979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚13737936047104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚13396699572832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13763658 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 703534518784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚61 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8444681 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 722809232896 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8454337 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8454337 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14441470565888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13737953086464 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3151995 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13396682533472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5318977 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8444681 % _timestamp‚1.7857936862430165e+09 _runtime‚ 360.320170646 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14441470565888 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚14494192139264 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14441575034880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5318979 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13737936047104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13771517774848 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 703621948416 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5446243 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚14494351911424 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚14098494826020 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 701812292548 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚799 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚13737953086464 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13771542678528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 703534518784 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13773316 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 722674364416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5318977 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5446512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5318979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚13737936047104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚13396699572832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13763658 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 703534518784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚61 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8444681 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 722809232896 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8454337 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚799 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚13755119849472 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8455204 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5325622 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 723712019456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 % _timestamp‚1.7857936866140609e+09 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5453071 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3155932 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8464874 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5453312 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚13788724891648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2297378 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 723577158144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13790496 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8464874 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 704414138368 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14459621424640 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5325620 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚14512302049792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8455204 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14459516948480 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚14512461814784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14459621424640 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 704501575168 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13788749795328 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 702775569006 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14116215314158 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13755102810112 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 704501575168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5325622 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 _step‚62 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241NÀ–„™…­M S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14459621424640 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 704501575168 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13788749795328 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 702775569006 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14116215314158 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13755102810112 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 704501575168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5325622 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 _step‚62 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13780824 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5325620 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13780824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚13413422705792 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 702689765770 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚799 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚13755119849472 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8455204 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5325622 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 723712019456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 % _timestamp‚1.7857936866140609e+09 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5453071 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3155932 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8464874 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5453312 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚13788724891648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2297378 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 723577158144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13790496 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8464874 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 704414138368 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14459621424640 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5325620 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚14512302049792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8455204 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14459516948480 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚14512461814784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251>'¢þ0†+ _wandbruntime‚360  _runtime‚360؃ü‡ö /loss/FRt1/detection_loss‚2.9807886842988496 .loss/FRt2/detection_loss‚6.320767888082908 $loss/total_loss‚77.9189781321298 /loss/FRt1/assignment_loss‚5.881112412509755 0loss/FRt2/assignment_loss‚5.4326904370234566 1loss/FBt2/assignment_loss‚0.47548049638668693 1loss/SRqqt1/detection_loss‚3.2516708766249405 .loss/FBt1/detection_loss‚34.30201858580112 $ _timestamp‚1.785793686615885e+09 _step‚63 2loss/SRqqt2/assignment_loss‚2.6534652871054574 /loss/FBt2/detection_loss‚10.463062124579787 trainer/global_step‚799 2loss/SRqqt1/assignment_loss‚2.8906791374601166 /loss/FBt1/assignment_loss‚0.729588907957077 0loss/SRqqt2/detection_loss‚2.537653294299637 epoch‚0 _runtime‚ 360.692679247;0&ˆú /loss/FRt1/detection_loss‚2.9807886842988496 .loss/FRt2/detection_loss‚6.320767888082908 $loss/total_loss‚77.9189781321298 /loss/FRt1/assignment_loss‚5.881112412509755 0loss/FRt2/assignment_loss‚5.4326904370234566 1loss/FBt2/assignment_loss‚0.47548049638668693 1loss/SRqqt1/detection_loss‚3.2516708766249405 .loss/FBt1/detection_loss‚34.30201858580112 $ _timestamp‚1.785793686615885e+09 _step‚63 2loss/SRqqt2/assignment_loss‚2.6534652871054574 /loss/FBt2/detection_loss‚10.463062124579787 trainer/global_step‚799 2loss/SRqqt1/assignment_loss‚2.8906791374601166 /loss/FBt1/assignment_loss‚0.729588907957077 0loss/SRqqt2/detection_loss‚2.537653294299637 epoch‚0 _runtime‚ 360.692679247?³¡‰‚(:„ ¤™ÄÓêýê¸ gpu.0.gpu‚23.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚121.08' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.36 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘ÀÓˆ>Š‚(:³  ¤™ÄÓ¿Þ°¹ disk.drbd1828.in‚0 disk.sda.out‚ 11.046875 disk.drbd2041.out‚0 network.sent‚757260 disk.drbd1226.out‚0 cpu‚14.55683155437153 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.sda.in‚0$ proc.memory.rssMB‚11253.30859375 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1584.in‚0 disk.sda1.in‚0* proc.memory.percent‚17.171186208724976 disk.drbd1510.out‚0 disk.rbd0.in‚0 disk.drbd2111.in‚0 disk.drbd1931.out‚0% memory_percent‚41.208893060684204$ disk./.usageGB‚570.3276138305664 disk.rbd0.out‚0 disk.drbd2118.out‚0 disk.dm-0.in‚0 disk.drbd1673.out‚0 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.drbd1965.in‚0* disk./.usagePercent‚31.895705802816575 disk.rbd1.out‚ 0.158203125 disk.drbd1550.in‚0 disk.drbd2111.out‚0 disk.drbd1673.in‚0 disk.drbd1285.out‚0 disk.sda2.out‚ 11.046875 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0# disk.nvme0n1.out‚13.09912109375 disk.drbd1828.out‚0 disk.drbd1552.out‚0 disk.sda2.in‚0 disk.drbd2078.out‚0 disk.drbd1931.in‚0* proc.memory.availableMB‚38529.33984375 proc.cpu.threads‚29 disk.drbd1550.out‚0 disk.nvme0n1.in‚0 disk.drbd1965.out‚0 disk.drbd1584.out‚0 disk.sda1.out‚0 network.recv‚98776 disk.drbd2078.in‚0 disk.drbd2118.in‚0 disk.dm-0.out‚13.09912109375 disk.drbd1938.out‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 Ô 0‹+ _wandbruntime‚382  _runtime‚382Ëý€fÆŒÀ ' lr-AdamW/pg1‚2.1880586326024876e-05 ' lr-AdamW/pg2‚2.1880586326024876e-05 trainer/global_step‚849 % _timestamp‚1.7857937087564754e+09 _runtime‚ 382.833458718 _step‚64ü-ìÊÄ ' lr-AdamW/pg1‚2.1880586326024876e-05 ' lr-AdamW/pg2‚2.1880586326024876e-05 trainer/global_step‚849 % _timestamp‚1.7857937087564754e+09 _runtime‚ 382.833458718 _step‚64@¹ôø0Ž+ _wandbruntime‚382  _runtime‚382üÔ$¥'ŸO bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 768018363904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8982339 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 745700756056 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5651127 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5787619 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5787875 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15343804851712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 745786559180 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8972683 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14233691149472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14596291236864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8972683 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14596274197504 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14233708188832 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5651129 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 747530654208 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14596291236864 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚14631873614848 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5651129 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15399916882432 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚14631898518528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 747618083840 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 747530654208 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5651127 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15343804851712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 747618083840 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5787875 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15343804851712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 745786559180 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8972683 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14233691149472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14596291236864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8972683 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14596274197504 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14233708188832 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5651129 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 747530654208 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14596291236864 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚14631873614848 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5651129 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15399916882432 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚14631898518528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 747618083840 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 747530654208 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5651127 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15343804851712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 747618083840 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 748497710592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14613457999872 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5657770 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2441844 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 768784224768 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 746578229278 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14650648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2441601 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14649105635328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5657772 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8992876 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5794383 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15361851234304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 748410273792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 KEDeviceStatsMonitor.on_t9‘ñ’rain_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚849 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7857937091318793e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14613440960512 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14613440960512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14997112393666 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _step‚66 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 748410273792 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15361955710464 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5657770 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 768916988928“|³&“­M _runtime‚ 383.209285681 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 748497710592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14613457999872 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5657770 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2441844 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 768784224768 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 746578229278 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14650648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2441601 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14649105635328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5657772 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8992876 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5794383 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15361851234304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 748410273792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚849 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7857937091318793e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14613440960512 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14613440960512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14997112393666 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _step‚66 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 748410273792 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15361955710464 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5657770 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 768916988928B¬à°0”+ _wandbruntime‚383  _runtime‚383Z›:ý•÷ 2loss/SRqqt1/assignment_loss‚2.9178962859701603 1loss/FBt2/assignment_loss‚0.37053499605368684 0loss/SRqqt1/detection_loss‚2.984864489591502 /loss/FBt2/detection_loss‚11.483107044533131 trainer/global_step‚849 .loss/FRt1/detection_loss‚2.867928923542301 .loss/FRt2/detection_loss‚6.101937231027855 % _timestamp‚1.7857937091340902e+09 /loss/FRt1/assignment_loss‚5.877691368261973 2loss/SRqqt2/assignment_loss‚2.6629811602879347 .loss/FBt1/detection_loss‚37.88615929914845 epoch‚0 _runtime‚ 383.210899268 _step‚67 /loss/FRt2/assignment_loss‚5.477126580364299 0loss/FBt1/assignment_loss‚0.7391372058126662 1loss/SRqqt2/detection_loss‚1.9124323631905595 %loss/total_loss‚81.28179694778451ÚlwΖû /loss/FRt1/assignment_loss‚5.877691368261973 2loss/SRqqt2/assignment_loss‚2.6629811602879347 .loss/FBt1/detection_loss‚37.88615929914845 epoch‚0 _runtime‚ 383.210899268 _step‚67 /loss/FRt2/assignment_loss‚5.477126580364299 0loss/FBt1/assignment_loss‚0.7391372058126662 1loss/SRqqt2/detection_loss‚1.9124323631905595 %loss/total_loss‚81.28179694778451 2loss/SRqqt1/assignment_loss‚2.9178962859701603 1loss/FBt2/assignment_loss‚0.37053499605368684 0loss/SRqqt1/detection_loss‚2.984864489591502 /loss/FBt2/detection_loss‚11.483107044533131 trainer/global_step‚849 .loss/FRt1/detection_loss‚2.867928923542301 .loss/FRt2/detection_loss‚6.101937231027855 % _timestamp‚1.7857937091340902e+09CÄLšqœ—‚(:‘ ³™ÄÓ–•ð¸ gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚120.742' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.24733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷3nŽD˜‚(:¹  ³™ÄÓÔ´º¹ disk.dm-0.in‚0 disk.drbd1356.in‚0 disk.sda2.out‚ 11.45703125 disk.drbd2078.out‚0 disk.drbd1550.in‚0 disk.drbd2111.in‚0 disk.drbd2118.in‚0$ proc.memory.rssMB‚11253.30859375$ disk./.usageGB‚570.3276176452637 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 11.45703125 proc.cpu.threads‚29 disk.rbd1.in‚0* proc.memory.availableMB‚38529.76953125 network.recv‚101448 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 network.sent‚802424 disk.nvme0n1.in‚0 disk.rbd0.in‚0# disk.nvme0n1.out‚13.38427734375 disk.dm-0.out‚13.38427734375 disk.drbd1673.out‚0 disk.drbd1817.out‚0* disk./.usagePercent‚31.895706016154417 disk.drbd1289.out‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0* proc.memory.percent‚17.171186208724976 disk.drbd1552.out‚0 disk.drbd1510.out‚0 disk.drbd1285.in‚0 disk.drbd2111.out‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 cpu‚14.522391515056494 disk.drbd1285.out‚0 disk.drbd1356.out‚0 disk.sda2.in‚0 disk.drbd1550.out‚0 disk.drbd1828.out‚0 disk.drbd2041.in‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1226.in‚0 disk.rbd1.out‚ 0.158203125 disk.drbd2118.out‚0% memory_percent‚41.208237409591675 disk.drbd2078.in‚0 disk.drbd1552.in‚0 disk.drbd1965.out‚0 disk.sda1.in‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.sda1.out‚0 disk.drbd1931.out‚0— Ö·0™+ _wandbruntime‚404  _runtime‚404û7Äš¾ ' lr-AdamW/pg1‚2.3169195650290183e-05 ' lr-AdamW/pg2‚2.3169195650290183e-05 trainer/global_step‚899 $ _timestamp‚1.785793730890247e+09 _runtime‚ 404.96724988 _step‚68ËÕm9ț $ _timestamp‚1.785793730890247e+09 _runtime‚ 404.96724988 _step‚68 ' lr-AdamW/pg1‚2.3169195650290183e-05 ' lr-AdamW/pg2‚2.3169195650290183e-05 trainer/global_step‚899D'): 0œ+ _wandbruntime‚404  _runtime‚404ZɌ٥'ŸO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15492254358528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15492229454848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6128420 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6128161 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 813257121280 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15860288985036 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15070699765472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚16305511479808 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3545697 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 791614219264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2582466 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5983279 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15454612347904 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5983279 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 789589219564 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 813122252800 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚15860391827520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚69 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚16246139137536 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3545695 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 791526789632 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5983277 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 % _timestamp‚1.7857937308913484e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚15454612347904 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 791526789632 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚15070716804832 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 404.968634514 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15483962 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15454629387264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9500685 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚16305351707648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 789675022688 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5983277 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16246139137536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024¨¬MÈÁž£O G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 789589219564 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 813122252800 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚15860391827520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚69 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚16246139137536 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3545695 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 791526789632 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5983277 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 % _timestamp‚1.7857937308913484e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚15454612347904 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 791526789632 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚15070716804832 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 A7DeviceStatsMonitor.on_train_batch_start/acti,¼yève.all.peak‚11887 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 404.968634514 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15483962 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15454629387264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9500685 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚16305351707648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 789675022688 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5983277 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16246139137536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15492254358528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15492229454848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6128420 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6128161 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 813257121280 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15860288985036 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15070699765472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚16305511479808 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3545697 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 791614219264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2582466 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5983279 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15454612347904 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5983279 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484EU¨PŸ‚(:… ™ÄÓú†Ž¹ gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚98.667' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚32.889 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0! @C ‚(:¸  ™ÄÓ¤ƒØ¹ disk.drbd1817.out‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚0 proc.cpu.threads‚29 disk.drbd1965.out‚0$ memory_percent‚41.20861887931824 network.recv‚104034 disk.drbd1965.in‚0 disk.sda2.out‚ 11.69921875 disk.drbd1550.in‚0 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd1226.in‚0 disk.drbd1356.in‚0 disk.drbd1289.out‚0 disk.sda.in‚0* proc.memory.percent‚17.171186208724976 cpu‚14.492781296695131* disk./.usagePercent‚31.895706016154417 disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0 disk.drbd1552.out‚0 disk.drbd1828.in‚0 disk.drbd1938.out‚0 disk.rbd1.out‚ 0.158203125 disk.drbd1226.out‚0 network.sent‚847439 disk.drbd1817.in‚0 disk.drbd1510.out‚0 disk.rbd0.out‚0 disk.drbd2078.in‚0 disk.drbd1510.in‚0# disk.nvme0n1.out‚13.52099609375 disk.rbd0.in‚0 disk.dm-0.out‚13.52099609375$ disk./.usageGB‚570.3276176452637 disk.sda2.in‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0* proc.memory.availableMB‚38529.51953125 disk.drbd1285.in‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚0 disk.drbd2078.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 11.69921875$ proc.memory.rssMB‚11253.30859375 disk.rbd1.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1673.in‚0Ìw‰0¡+ _wandbruntime‚405  _runtime‚405H“ðH¯&¢©M VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚15471779110912 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚15471796150272 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9520878 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6135204 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚16323467239936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2585316 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15510800 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 814030668288 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 792493846016 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3549632 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15877906630578 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3549634 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚15087456977152 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5989920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚16264289996288 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 790466692786 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15501128 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9511208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 814159238144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚70 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15510800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5989922 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚16323620713472 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15878009473174 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6134948 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15509436571648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5989920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚16264289996288 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 790466692786 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15501128 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9511208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 814159238144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚70 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15510800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5989922 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚16323620713472 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15878009473174 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6134948 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15509436571648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2585316 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15510800 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 814030668288 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 792493846016 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3549632 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15877906630578 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3549634 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚15087456977152 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17148473423360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16312967537664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚17148473423360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16352585294848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17211109753344 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _runtime‚ 427.360066716 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3742547 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10038343 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3742545 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16741288907028 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16344114 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 858499554816 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6315427 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 835610354688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 833563486196 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 835610354688 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 835522925056 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2727401 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10038343 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚16741186064544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 % _timestamp‚1.7857937532828617e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15907708381472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10028687 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚15907725420832 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚17148577892352 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16312950498304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6469946 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032? ”­'®§O trainer/global_step‚949 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6315427 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 858364686336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2727646 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚17210949981184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17148473423360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16312967537664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚17148473423360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16352585294848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17211109753344 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _runtime‚ 427.360066716 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3742547 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10038343 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3742545 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16741288907028 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16344114 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 858499554816 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6315427 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 835610354688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 833563486196 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 835610354688 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 835522925056 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2727401 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10038343 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚16741186064544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 % _timestamp‚1.7857937532828617e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15907708381472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10028687 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚15907725420832 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚17148577892352 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16312950498304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6469946 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚17148577892352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16312967537664 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16353772 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 833477683072Iª xº0¯+ _wandbruntime‚427  _runtime‚4272Lyû²&°¬M `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 859401726976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16330117261312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3746484 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2730506 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 trainer/global_step‚949 $ _timestamp‚1.785793753652627e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16330117261312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15924448553792 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6322070 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3746482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚16758803710086 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚17166624282112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6322072 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17229061374464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16758906552682 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16370952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16369817315328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10039210 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 836402544640 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6476990 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10048880 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 859268962816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 _step‚74 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚17166624282112 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6322070 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 836489981440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10039210 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 836489981440 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 836402544640 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10048880 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 834440959530 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 834355156294 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6476741 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16369792411648 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6322072 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17229061374464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16758906552682 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16370952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16369817315328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10039210 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 836402544640 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6476990 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10048880 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 859268962816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 _step‚74 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚17166624282112 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6322070 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 836489981440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10039210 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 836489981440 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 836402544640 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10048880 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 834440959530 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 834355156294 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6476741 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16369792411648 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16330117261312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3746484 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2730506 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247J¥Ò›q0²+ _wandbruntime‚427  _runtime‚427´ ­Xú³ô 2loss/SRqqt2/assignment_loss‚2.3704478469076036 0loss/FBt1/assignment_loss‚0.7090849302088221 .loss/FBt2/detection_loss‚6.064447963370366 epoch‚0 .loss/FRt1/assignment_loss‚5.69036587760562 1loss/FBt2/assignment_loss‚0.11674298005444662 .loss/FRt1/detection_loss‚3.311788015280451 0loss/SRqqt2/detection_loss‚2.799556705060298 /loss/FBt1/detection_loss‚27.769914756218594 %loss/total_loss‚66.38165261874185 trainer/global_step‚949 _step‚75 $ _timestamp‚1.785793753654427e+09 1loss/SRqqt1/assignment_loss‚2.791909765411209 .loss/FRt2/detection_loss‚5.909915473726061 1loss/SRqqt1/detection_loss‚3.4372618939435866 _runtime‚ 427.731198624 /loss/FRt2/assignment_loss‚5.410216410954793)䥵þ´ø /loss/FRt2/assignment_loss‚5.410216410954793 2loss/SRqqt2/assignment_loss‚2.3704478469076036 0loss/FBt1/assignment_loss‚0.7090849302088221 .loss/FBt2/detection_loss‚6.064447963370366 epoch‚0 .loss/FRt1/assignment_loss‚5.69036587760562 1loss/FBt2/assignment_loss‚0.11674298005444662 .loss/FRt1/detection_loss‚3.311788015280451 0loss/SRqqt2/detection_loss‚2.799556705060298 /loss/FBt1/detection_loss‚27.769914756218594 %loss/total_loss‚66.38165261874185 trainer/global_step‚949 _step‚75 $ _timestamp‚1.785793753654427e+09 1loss/SRqqt1/assignment_loss‚2.791909765411209 .loss/FRt2/detection_loss‚5.909915473726061 1loss/SRqqt1/detection_loss‚3.4372618939435866 _runtime‚ 427.731198624KñÇ–F?µ‚(:´  à™ÄÓö’¦» network.recv‚108539 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.drbd1965.in‚0 disk.sda2.in‚0 disk.drbd2111.out‚0 disk.dm-0.out‚14.36376953125' proc.memory.availableMB‚ 38529.84375 disk.drbd1285.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.sda.out‚ 12.33203125 cpu‚14.435066081555801 disk.drbd1552.out‚0 disk.drbd1289.in‚0 proc.cpu.threads‚29 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd2041.out‚0 disk.sda1.out‚0 disk.drbd1931.in‚0# memory_percent‚41.2081241607666 disk.drbd1510.in‚0 disk.rbd0.out‚0 disk.drbd2078.in‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.drbd1938.in‚0 disk.drbd1817.in‚0 disk.rbd1.out‚ 0.158203125 disk.drbd1550.in‚0 disk.dm-0.in‚0 disk.drbd1673.in‚0 disk.drbd1226.in‚0 disk.sda.in‚0 disk.sda1.in‚0$ proc.memory.rssMB‚11253.30859375 disk.drbd1938.out‚0 disk.drbd1510.out‚0# disk.nvme0n1.out‚14.36376953125 disk.rbd0.in‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.drbd1226.out‚0 disk.rbd1.in‚0* disk./.usagePercent‚31.895709429559872 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.drbd1584.in‚0* proc.memory.percent‚17.171186208724976$ disk./.usageGB‚570.3276786804199 disk.drbd2078.out‚0 network.sent‚916494 disk.drbd1965.out‚0 disk.drbd1673.out‚0 disk.nvme0n1.in‚0 disk.drbd1285.in‚0 disk.sda2.out‚ 12.33203125 disk.drbd1931.out‚0 disk.drbd1552.in‚0^qÜ œ¶‚(:‘ à™ÄÓû¼·» gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚96.344' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚32.114666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×óËÇ0·+ _wandbruntime‚449  _runtime‚449¯ïkŸ¿ ' lr-AdamW/pg1‚2.5746414298820792e-05 ' lr-AdamW/pg2‚2.5746414298820792e-05 trainer/global_step‚999 $ _timestamp‚1.785793775883308e+09 _runtime‚ 449.960288182 _step‚76'æ„ɹà ' lr-AdamW/pg1‚2.5746414298820792e-05 ' lr-AdamW/pg2‚2.5746414298820792e-05 trainer/global_step‚999 $ _timestamp‚1.785793775883308e+09 _runtime‚ 449.960288182 _step‚76LÍ„ -0º+ _wandbruntime‚449  _runtime‚449“ô6©'»£O ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 903550050304 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 trainer/global_step‚999 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6647577 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6647579 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3939397 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚17622083144052 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 903687015936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚17171305688064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 879606490112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18050912178176 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10556689 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 879519060480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚17622185986536 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18050807709184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17213924 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16744734036832 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6647579 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3939395 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚18116653054464 _step‚77 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10566345 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 877451949704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10556689 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚17171288648704 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 879519060480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚18050912178176 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚17171305688064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18116491185152 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18050807709184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2870460 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 % _timestamp‚1.7857937758843884e+09 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 _runtime‚ 449.961663216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6647577 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6809594 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16744716997472 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 877366146580 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2870199 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚17212966038528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10566345 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6809857 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 879606490112 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚17171288648704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172129411348488EïQ&¼§O OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6647579 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3939395 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚18116653054464 _step‚77 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10566345 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 877451949704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10556689 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚17171288648704 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 879519060480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚18050912178176 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚17171305688064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18116491185152 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18050807709184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2870460 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 % _timestamp‚1.7857937758843884e+09 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 _runtime‚ 449.961663216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6647577 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6809594 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16744716997472 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 877366146580 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2870199 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚17212966038528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10566345 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6809857 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 879606490112 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚17171288648704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚17212941134848 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 903550050304 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 trainer/global_step‚999 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6647577 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6647579 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3939397 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚17622083144052 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 903687015936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚17171305688064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 879606490112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18050912178176 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10556689 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 879519060480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚17622185986536 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18050807709184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17213924 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.ų•œ\freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16744734036832 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121MÝ„¦›½‚(: ï™ÄÓäÞ›¹ gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚123.35' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.11666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸ôWÇ<¾‚(:±  ï™ÄÓœ‚ã¹ disk.drbd2118.in‚0 disk.sda.out‚ 12.6953125 disk.drbd1938.in‚0$ memory_percent‚41.20892882347107* proc.memory.percent‚17.171186208724976 disk.drbd1356.in‚0 disk.drbd1817.in‚0* proc.memory.availableMB‚38529.31640625 disk.drbd1285.out‚0 disk.drbd2111.in‚0 disk.drbd2118.out‚0 disk.sda1.out‚0$ proc.memory.rssMB‚11253.30859375 proc.cpu.threads‚29$ disk./.usageGB‚570.3276786804199 disk.drbd1931.in‚0 disk.drbd1550.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd2041.in‚0 disk.drbd1931.out‚0 network.sent‚961512* disk./.usagePercent‚31.895709429559872 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.sda1.in‚0! disk.nvme0n1.out‚ 15.064453125 disk.drbd1356.out‚0 disk.drbd1552.in‚0 disk.drbd2078.in‚0 disk.drbd1550.in‚0 disk.drbd1285.in‚0 disk.dm-0.out‚ 15.064453125 disk.drbd1673.out‚0 disk.drbd1817.out‚0 disk.rbd1.out‚ 0.158203125 disk.drbd1828.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 network.recv‚111089 disk.rbd0.in‚0 disk.sda.in‚0 disk.dm-0.in‚0 disk.rbd0.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 cpu‚14.41115611937486 disk.drbd1965.out‚0 disk.sda2.in‚0 disk.drbd1226.out‚0 disk.nvme0n1.in‚0 disk.sda2.out‚ 12.6953125‹á0¿+ _wandbruntime‚450  _runtime‚450<ö²&À¬M H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2873029 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚78 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17231104 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18068958567936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚17188472451072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6654220 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 880398680064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10576882 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17221432 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚17639700789594 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6654220 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18068854091776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3943334 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10567212 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚17188455411712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚17230173155328 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 880398680064 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3943332 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18134604741120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6816613 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 878329423038 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 $ _timestamp‚1.785793776255971e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚17230148251648 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚17639803632190 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 880486116864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚17188455411712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 904589253632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚999 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 904456489472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 880486116864 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6816361 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16761474209152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 _runtime‚ 450.333313024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2873279 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6654222 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6654222 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251ƒD•¶&Á°M G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17231104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17221432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 _runtime‚ 450.333313024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2873279 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6654222 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6654222 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2873029 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚78 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17231104 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18068958567936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚17188472451072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6654220 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 880398680064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10576882 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17221432 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚17639700789594 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6654220 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18068854091776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3943334 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10567212 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚17188455411712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚17230173155328 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 880398680064 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3943332 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18134604741120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6816613 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 878329423038 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 $ _timestamp‚1.785793776255971e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚17230148251648 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚17639803632190 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 880486116864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚17188455411712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 904589253632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚999 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 904456489472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 880486116864 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6816361 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16761474209152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18953141995008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18953246464000 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚18502980223560 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6979729 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 923602625536 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚17581725613472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 921340413212 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11094347 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3013478 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 923515195904 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 923515195904 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.lar_–±ège_pool.peak‚ 2768240640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚18029626799104 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18074076 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11084691 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1049 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18953141995008 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚18503083066044 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11094347 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18073321878528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚18029643838464 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6979727 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4136247 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18073296974848 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6979727 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18064418 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 948841387008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 921254610088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 _step‚81 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18953141995008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18953246464000 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚18502980223560 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6979729 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 923602625536 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚17581725613472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 921340413212 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11094347 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3013478 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 923515195904 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 923515195904 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚18029626799104 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18074076 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11084691 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1049 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18953141995008 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚18503083066044 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11094347 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18073321878528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚18029643838464 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6979727 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4136247 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18073296974848 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6979727 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0Qq±J¶0Í+ _wandbruntime‚472  _runtime‚472N K´&ήM [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19040271587840 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚18046810601472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6986372 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11104884 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 949742592512 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚17598465785792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18081584 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7156462 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 924482252288 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚18520700711698 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4140184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3016026 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19040113920000 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚18046810601472 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6986370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚ 472.347690031 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6986372 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 924394815488 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6986370 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚18971292853760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 922217886546 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18971292853760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18091256 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚18520597869102 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3016278 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 949609828352 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 _step‚82 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 924394815488 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7156208 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18090504091648 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚17598482825152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18091256 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18971188377600 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392£Ô¯¿c"ϲM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11095214 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3016278 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 949609828352 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 _step‚82 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 924394815488 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7156208 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18090504091648 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚17598482825152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18091256 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18971188377600 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19040271587840 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚18046810601472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6986372 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11104884 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 949742592512 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚17598465785792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18081584 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7156462 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 924482252288 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚18520700711698 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4140184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3016026 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19040113920000 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚18046810601472 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6986370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚ 472.347690031 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6986372 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 924394815488 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6986370 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚18971292853760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 922217886546 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18971292853760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18091256 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1852059786910tåxU2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EÇÖÁ _runtime‚ 494.222200618 _step‚84 ' lr-AdamW/pg1‚2.8323632947351405e-05 ' lr-AdamW/pg2‚2.8323632947351405e-05 trainer/global_step‚1099 % _timestamp‚1.7857938201452281e+09H>@ÙË×Å _step‚84 ' lr-AdamW/pg1‚2.8323632947351405e-05 ' lr-AdamW/pg2‚2.8323632947351405e-05 trainer/global_step‚1099 % _timestamp‚1.7857938201452281e+09 _runtime‚ 494.222200618TðGs0Ø+ _wandbruntime‚494  _runtime‚494ˤÖͪ'Ù¤O WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3157777 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 965143073596 trainer/global_step‚1099 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19383980145552 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚19855476280832 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚18887964949504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 967511331328 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚19855476280832 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚19855580749824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7490874 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11612693 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18934228 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3157526 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 994071635968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 % _timestamp‚1.7857938201463163e+09 _runtime‚ 494.223512622 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7311877 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 965228876720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚18418751268832 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 993940961792 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 967511331328 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11622349 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚19383877303068 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _step‚85 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚19855580749824 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7311879 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18933677718528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7311879 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11622349 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7490621 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19927593776640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0’k®'Ú¨O VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 967511331328 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11622349 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚19383877303068 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _step‚85 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚19855580749824 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7311879 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18933677718528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7311879 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11622349 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7490621 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19927593776640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3157777 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 965143073596 trainer/global_step‚1099 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19383980145552 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚19855476280832 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚18887964949504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 967511331328 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚19855476280832 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚19855580749824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7490874 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11612693 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18934228 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3157526 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 994071635968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 % _timestamp‚1.7857938201463163e+09 _runtime‚ 494.223512622 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7311877 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 965228876720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚18418751268832 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 993940961792 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658U´íÌÈ0Û+ _wandbruntime‚494  _runtime‚494DŸCÉÜ®M C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 994973826560 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18941736 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7318522 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚19873627139584 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚19873522663424 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18435474401792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3160375 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚18905131712512 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 968478387712 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚19873627139584 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 % _timestamp‚1.7857938205245855e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7318520 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7497653 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4337034 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 966106350054 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚18905148751872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7497407 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4337032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3160619 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19401494948610 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18951408 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚18905131712512 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7318522 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 966020546818 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11632886 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 968390950912 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18951408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚18905148751872 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 968478387712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0Dx¸&ݲM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18950859931648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 994843159552 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18950884835328 _runtime‚ 494.601940067 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19945703091200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7318520 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7497407 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4337032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3160619 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19401494948610 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18951408 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚18905131712512 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7318522 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 966020546818 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11632886 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 968390950912 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18951408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚18905148751872 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 968478387712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 994973826560 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18941736 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7318522 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚19873627139584 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚19873522663424 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18435474401792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3160375 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚18905131712512 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 968478387712 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚19873627139584 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 % _timestamp‚1.7857938205245855e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7318520 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7497653 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4337034 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 966106350054 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚18905148751872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ODeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20757810566656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚20264877225060 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1009031537104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚20833113201664 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7831626 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7831365 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1011594896384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1009117340228 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1011594896384 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 trainer/global_step‚1149 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12140695 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1039243609600 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 516.24764 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7644027 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚19794008654848 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20757810566656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4529947 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12140695 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1039243609600 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 516.24764 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7644027 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚19794008654848 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20757810566656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4529947 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20757810566656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚20264877225060 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1009031537104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚20833113201664 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7831626 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7831365 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1011594896384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1009117340228 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1011594896384 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 trainer/global_step‚1149 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7857938425308228e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19811560 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7650670 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12151218 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3304589 _runtime‚ 516.608226776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚19763469862912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚19811215771648 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1012474523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1040143676416 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7838473 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7650672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19801888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4533884 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1009994813562 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚19763486902272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1012474523136 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20775961425408 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19811560 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4533882 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19763469862912 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚20851235072512 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12151218 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 trainer/global_step‚1149 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19801888 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20775856949248 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7838210 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1040019300864 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7650670 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1009909010326 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1012387086336 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12160888 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚20282392028118 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7838210 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1040019300864 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7650670 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1009909010326 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1012387086336 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12160888 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚20282392028118 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7857938425308228e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19811560 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7650670 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12151218 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3304589 _runtime‚ 516.608226776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚19763469862912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚19811215771648 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1012474523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1040143676416 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7838473 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7650672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19801888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4533884 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1009994813562 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚19763486902272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1012474523136 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20775961425408 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19811560 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4533882 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19763469862912 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚20851235072512 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12151218 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 trainer/global_step‚1149 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19801888 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20775856949248 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160Zõäîí0î+ _wandbruntime‚516  _runtime‚516š<¶Aýï÷ /loss/FRt2/assignment_loss‚5.291435935918023 .loss/FBt2/detection_loss‚4.483948855885205 % _timestamp‚1.7857938425328825e+09 _step‚91 0loss/FBt1/assignment_loss‚0.6202493260021914 .loss/FRt2/detection_loss‚4.560547227806905 1loss/SRqqt1/detection_loss‚2.4924564853284035 epoch‚0 trainer/global_step‚1149 _runtime‚ 516.609658355 /loss/FRt1/assignment_loss‚5.308324116926927 2loss/SRqqt1/assignment_loss‚2.6494941596399273 /loss/FBt2/assignment_loss‚0.311552050923451 /loss/FRt1/detection_loss‚2.5064393769089994 1loss/SRqqt2/detection_loss‚2.4556734175295625 %loss/total_loss‚63.02633429733911 2loss/SRqqt2/assignment_loss‚2.3599192208713955 .loss/FBt1/detection_loss‚29.98629412359812¿Á_Òðû /loss/FRt2/assignment_loss‚5.291435935918023 .loss/FBt2/detection_loss‚4.483948855885205 % _timestamp‚1.7857938425328825e+09 _step‚91 0loss/FBt1/assignment_loss‚0.6202493260021914 .loss/FRt2/detection_loss‚4.560547227806905 1loss/SRqqt1/detection_loss‚2.4924564853284035 epoch‚0 trainer/global_step‚1149 _runtime‚ 516.609658355 /loss/FRt1/assignment_loss‚5.308324116926927 2loss/SRqqt1/assignment_loss‚2.6494941596399273 /loss/FBt2/assignment_loss‚0.311552050923451 /loss/FRt1/detection_loss‚2.5064393769089994 1loss/SRqqt2/detection_loss‚2.4556734175295625 %loss/total_loss‚63.02633429733911 2loss/SRqqt2/assignment_loss‚2.3599192208713955 .loss/FBt1/detection_loss‚29.98629412359812[ûö(§ñ‚(:’ ºšÄÓŠ½ì¸ gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚124.954' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚41.651333333333326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íôèHò‚(:½  ºšÄÓ㦮¹ disk.sda1.in‚0 disk.rbd1.in‚0 disk.drbd1584.in‚0 network.recv‚122491 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1931.in‚0 disk.drbd1673.out‚0 disk.sda1.out‚0 network.sent‚ 1.124006e+06 disk.drbd1817.out‚0# disk.nvme0n1.out‚16.77587890625 disk.drbd1552.out‚0 disk.drbd2078.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 proc.cpu.threads‚29 disk.drbd1550.in‚0 disk.drbd2111.out‚0 disk.dm-0.out‚16.77587890625$ proc.memory.rssMB‚11253.30859375 disk.drbd1938.in‚0 disk.drbd1673.in‚0 disk.drbd1552.in‚0 disk.drbd1510.out‚0 disk.sda2.in‚0 disk.drbd1828.out‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd2078.in‚0 disk.dm-0.in‚0* proc.memory.availableMB‚38529.49609375$ disk./.usageGB‚570.3259773254395 disk.drbd1356.out‚0 disk.drbd1965.out‚0 disk.rbd1.out‚ 0.158203125 disk.drbd2111.in‚0 disk.drbd1828.in‚0 cpu‚14.324630836602113 disk.drbd1550.out‚0# memory_percent‚41.2086546421051 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.sda.in‚0 disk.drbd1510.in‚0 disk.nvme0n1.in‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0* proc.memory.percent‚17.171186208724976* disk./.usagePercent‚31.895614280882857 disk.drbd1285.out‚0 disk.drbd1356.in‚0 disk.sda2.out‚ 15.13671875 disk.sda.out‚ 15.13671875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0e0­0ó+ _wandbruntime‚538  _runtime‚538þçí;Åô¿ & lr-AdamW/pg2‚3.090085159588201e-05 trainer/global_step‚1199 % _timestamp‚1.7857938641298695e+09 _runtime‚ 538.206854024 _step‚92 & lr-AdamW/pg1‚3.090085159588201e-05@ï—}Éõà _runtime‚ 538.206854024 _step‚92 & lr-AdamW/pg1‚3.090085159588201e-05 & lr-AdamW/pg2‚3.090085159588201e-05 trainer/global_step‚1199 % _timestamp‚1.7857938641298695e+09\Õ5ܺ0ö+ _wandbruntime‚538  _runtime‚538"æØ«¥÷¬O C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1055591031808 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 trainer/global_step‚1199 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7976179 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚93 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12668697 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20654389398528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TADeviceStatsMonitor.on_train_batch_start/al©Y9 located_bytes.all.freed‚21660144852480 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚21145671462084 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20644874 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1084455763456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚20604641250304 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20604641250304 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20654364494848 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1053005803736 % _timestamp‚1.7857938641309602e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1055591031808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21738683292672 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1052920000612 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚20604658289664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1084318797824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8172985 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12678353 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1055503602176 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚21145774304568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4726795 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20092768500832 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3446188 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚21660144852480 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚20604658289664 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7976177 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4726797 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 _runtime‚ 538.208230123 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7976179 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1055503602176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0}‰¶'ø°O SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7976179 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1055503602176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1055591031808 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 trainer/global_step‚1199 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7976179 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚93 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12668697 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20654389398528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚21660144852480 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚21145671462084 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20644874 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1084455763456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚20604641250304 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20604641250304 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20654364494848 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1053005803736 % _timestamp‚1.7857938641309602e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1055591031808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21738683292672 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1052920000612 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚20604658289664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1084318797824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8172985 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12678353 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1055503602176 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚21145774304568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4726795 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20092768500832 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3446188 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚21660144852480 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚20604658289664 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7976177 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4726797 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 _runtime‚ 538.208230123]Tè‚0ù+ _wandbruntime‚538  _runtime‚5387ªí¼&ú¶M YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚20621808013312 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚20621825052672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20109508673152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7857938644995792e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7982820 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚21678295711232 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168153600 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1085215635968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1056470658560 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚20621825052672 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚20621808013312 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21756787247616 _step‚94 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7982822 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7982820 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 _runtime‚ 538.576963119 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12679220 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚21163289107626 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7982822 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚21678295711232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚21678191235072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1085358885888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20671712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8179770 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8179508 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1053797473834 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121^idµ.ûºM MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7982822 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7982820 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 _runtime‚ 538.576963119 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12679220 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚21163289107626 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7982822 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚21678295711232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚21678191235072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1085358885888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20671712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8179770 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8179508 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1053797473834 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚20621808013312 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚20621825052672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20109508673152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7857938644995792e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7982820 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚21678295711232 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168153600 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1085215635968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1056470658560 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚20621825052672 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚20621808013312 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21756787247616 _step‚94^µt•0ü+ _wandbruntime‚538  _runtime‚538€C&?þýø /loss/FRt1/assignment_loss‚5.387250957489013 /loss/FRt2/assignment_loss‚5.389213648709384 2loss/SRqqt1/assignment_loss‚2.5577415972948074 1loss/SRqqt2/assignment_loss‚2.228946098914513 0loss/SRqqt1/detection_loss‚3.198324588857455 epoch‚0 _step‚95 0loss/FBt1/assignment_loss‚0.4092370261039053 1loss/SRqqt2/detection_loss‚1.9128849916406683 /loss/FBt1/detection_loss‚23.236292652918824 .loss/FBt2/detection_loss‚6.009364910343822 trainer/global_step‚1199 $ _timestamp‚1.785793864501512e+09 _runtime‚ 538.578314398 .loss/FRt2/detection_loss‚5.726067664948377 &loss/total_loss‚59.579310429950986 1loss/FBt2/assignment_loss‚0.13586928407395524 /loss/FRt1/detection_loss‚3.3881170086562635ýõþü 1loss/FBt2/assignment_loss‚0.13586928407395524 /loss/FRt1/detection_loss‚3.3881170086562635 /loss/FRt1/assignment_loss‚5.387250957489013 /loss/FRt2/assignment_loss‚5.389213648709384 2loss/SRqqt1/assignment_loss‚2.5577415972948074 1loss/SRqqt2/assignment_loss‚2.228946098914513 0loss/SRqqt1/detection_loss‚3.198324588857455 epoch‚0 _step‚95 0loss/FBt1/assignment_loss‚0.4092370261039053 1loss/SRqqt2/detection_loss‚1.9128849916406683 /loss/FBt1/detection_loss‚23.236292652918824 .loss/FBt2/detection_loss‚6.009364910343822 trainer/global_step‚1199 $ _timestamp‚1.785793864501512e+09 _runtime‚ 538.578314398 .loss/FRt2/detection_loss‚5.726067664948377 &loss/total_loss‚59.579310429950986_Ão,öœÿ‚(:‘ ÉšÄÓ§¾ê¸ gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚117.163' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.05433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~ýÐ?€‚(:´  ÉšÄÓÝ›²¹$ disk./.usageGB‚570.3269958496094 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0$ memory_percent‚41.20972156524658 disk.drbd1356.in‚0 network.sent‚ 1.169312e+06 disk.drbd1938.in‚0 disk.rbd1.out‚ 0.158203125 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd1584.in‚0 disk.nvme0n1.out‚ 17.19140625 disk.drbd2041.in‚0 disk.sda1.in‚0 disk.drbd1552.in‚0$ proc.memory.rssMB‚11253.30859375 disk.drbd1226.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.drbd1965.out‚0 disk.rbd0.out‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1965.in‚0 network.recv‚125357* disk./.usagePercent‚31.895671242086358 disk.drbd1817.out‚0 disk.nvme0n1.in‚0 disk.drbd2111.in‚0 disk.dm-0.out‚ 17.19140625 disk.drbd1510.in‚0 disk.drbd1289.in‚0 disk.drbd1550.in‚0 proc.cpu.threads‚29 disk.sda.in‚0 disk.sda.out‚ 15.4609375 disk.drbd2041.out‚0 disk.drbd1828.out‚0* proc.memory.percent‚17.171186208724976 disk.drbd1938.out‚0 disk.drbd1673.out‚0 disk.sda2.in‚0 disk.sda2.out‚ 15.4609375 disk.drbd2078.out‚0 disk.drbd2118.out‚0 disk.drbd1510.out‚0 disk.drbd1285.out‚0 disk.drbd2078.in‚0 disk.drbd1550.out‚0 disk.dm-0.in‚0 disk.drbd1673.in‚0 disk.drbd1285.in‚0 disk.drbd1356.out‚0( proc.memory.availableMB‚ 38528.796875 cpu‚14.306096336733908ù94»‘‚(:† ØšÄÓ¨Ù÷¸ gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚123.714' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚41.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àM ->‚‚(:³  ØšÄÓíÏÁ¹ disk.drbd1552.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.drbd2041.out‚0 disk.sda1.out‚0 disk.drbd1931.in‚0* disk./.usagePercent‚31.895695135924534 disk.drbd1965.in‚0 disk.drbd1510.out‚0 proc.cpu.threads‚29 disk.nvme0n1.out‚ 17.54296875 disk.dm-0.in‚0* proc.memory.percent‚17.171186208724976 disk.rbd1.in‚0 disk.drbd2078.out‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd1550.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 network.recv‚127804 disk.drbd1510.in‚0 disk.rbd0.out‚0 disk.drbd2078.in‚0 disk.drbd2118.in‚0 disk.sda.out‚ 15.890625 disk.drbd2041.in‚0 disk.sda1.in‚0 disk.drbd1356.in‚0 disk.rbd0.in‚0 cpu‚14.28885382549635 disk.drbd1673.out‚0 network.sent‚ 1.214372e+06 disk.drbd1552.in‚0 disk.sda2.out‚ 15.890625$ proc.memory.rssMB‚11253.30859375 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.drbd1584.out‚0 disk.sda.in‚0 disk.drbd1931.out‚0* proc.memory.availableMB‚38528.54296875 disk.drbd1673.in‚0 disk.drbd1226.out‚0 disk.nvme0n1.in‚0 disk.drbd1285.out‚0 disk.drbd1828.in‚0 disk.drbd2118.out‚0$ disk./.usageGB‚570.3274230957031 disk.drbd1817.in‚0 disk.drbd1285.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 0.158203125 disk.sda2.in‚0 disk.dm-0.out‚ 17.54296875$ memory_percent‚41.21010899543762 disk.drbd1938.in‚0 disk.drbd1938.out‚0p`†d0ƒ+ _wandbruntime‚560  _runtime‚560,ˆ®3Ä„¾ trainer/global_step‚1249 % _timestamp‚1.7857938866735425e+09 _runtime‚ 560.75051743 _step‚96 & lr-AdamW/pg1‚3.218946092014731e-05 & lr-AdamW/pg2‚3.218946092014731e-05O ÿÈ…Â & lr-AdamW/pg2‚3.218946092014731e-05 trainer/global_step‚1249 % _timestamp‚1.7857938866735425e+09 _runtime‚ 560.75051743 _step‚96 & lr-AdamW/pg1‚3.218946092014731e-05`À@js0†+ _wandbruntime‚560  _runtime‚560¿Üä˲'‡¬O G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 % _timestamp‚1.7857938866746213e+09 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20929760077472 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4923645 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22562479138304 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1096894267244 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22562583607296 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1099499737600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚21462996440064 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3589015 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22644172211712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21514684 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 trainer/global_step‚1249 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8308329 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13206355 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8512918 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22562479138304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _runtime‚ 560.751917373 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8512660 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8308329 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚21514745238528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1129584648192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1096808464120 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8308327 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 _step‚97 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20929777116832 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4923647 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚21462979400704 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1099587167232 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22562583607296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8308327 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚21462996440064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121âëO¶'ˆ°O PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13196699 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22562479138304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _runtime‚ 560.751917373 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8512660 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8308329 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚21514745238528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1129584648192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1096808464120 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8308327 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 _step‚97 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20929777116832 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4923647 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚21462979400704 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1099587167232 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22562583607296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8308327 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚21462996440064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 % _timestamp‚1.7857938866746213e+09 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20929760077472 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4923645 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22562479138304 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1096894267244 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22562583607296 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1099499737600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚21462996440064 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3589015 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22644172211712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21514684 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 trainer/global_step‚1249 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8308329 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13206355 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8512918 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚98 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13207222 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21531864 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8519427 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1130355514368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1100379357184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13207222 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8314970 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚21480163203072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21522192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4927582 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22662282966016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22580629997056 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚21480163203072 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8314972 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1100466793984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21531864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚21480146163712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4927584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22580629997056 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚21480146163712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3591845 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22580525520896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13216892 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1100379357184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7857938870453804e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13207222 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21531864 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8519427 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1130355514368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1100379357184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13207222 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8314970 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚21480163203072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21522192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4927582 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22662282966016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22580629997056 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚21480163203072 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8314972 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1100466793984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21531864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚21480146163712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4927584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22580629997056 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚21480146163712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3591845 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22580525520896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13216892 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1100379357184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7857938870453804e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚98 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0bn±Wü0Œ+ _wandbruntime‚561  _runtime‚561‡…“þø /loss/FRt1/assignment_loss‚5.142946846008301 /loss/FBt1/detection_loss‚26.728649301609646 _runtime‚ 561.123914133 0loss/FRt2/assignment_loss‚5.4322024413517545 2loss/SRqqt2/assignment_loss‚2.3621962865193686 /loss/SRqqt1/detection_loss‚2.53720731036776 1loss/SRqqt2/detection_loss‚2.3022015065607597 %loss/total_loss‚64.17094918081503 trainer/global_step‚1249 % _timestamp‚1.7857938870471299e+09 /loss/FBt1/assignment_loss‚0.683220992796123 1loss/FBt2/assignment_loss‚0.22661307680342968 /loss/FRt1/detection_loss‚2.7347536822557448 .loss/FRt2/detection_loss‚5.622208167931864 .loss/FBt2/detection_loss‚7.768094203440104 _step‚99 2loss/SRqqt1/assignment_loss‚2.6306553651701727 epoch‚0°HŽü trainer/global_step‚1249 % _timestamp‚1.7857938870471299e+09 /loss/FBt1/assignment_loss‚0.683220992796123 1loss/FBt2/assignment_loss‚0.22661307680342968 /loss/FRt1/detection_loss‚2.7347536822557448 .loss/FRt2/detection_loss‚5.622208167931864 .loss/FBt2/detection_loss‚7.768094203440104 _step‚99 2loss/SRqqt1/assignment_loss‚2.6306553651701727 epoch‚0 /loss/FRt1/assignment_loss‚5.142946846008301 /loss/FBt1/detection_loss‚26.728649301609646 _runtime‚ 561.123914133 0loss/FRt2/assignment_loss‚5.4322024413517545 2loss/SRqqt2/assignment_loss‚2.3621962865193686 /loss/SRqqt1/detection_loss‚2.53720731036776 1loss/SRqqt2/detection_loss‚2.3022015065607597 %loss/total_loss‚64.17094918081503cÉr‚(:’ çšÄÓ‡ˆ¹ gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚121.174' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.391333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«õB‚(:·  çšÄÓÑÖй disk.rbd0.out‚0 disk.drbd1817.out‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 38528.71875) disk./.usagePercent‚31.89570302942465 disk.drbd1817.in‚0 disk.nvme0n1.in‚0 disk.sda2.out‚ 16.48828125 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd1673.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 proc.cpu.threads‚29 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd2111.out‚0 disk.sda.out‚ 16.48828125 disk.drbd1356.out‚0$ proc.memory.rssMB‚11253.30859375 disk.drbd1289.out‚0 disk.drbd1550.in‚0 disk.drbd1931.in‚0 disk.drbd1226.in‚0* proc.memory.percent‚17.171186208724976 disk.drbd2078.out‚0$ memory_percent‚41.20984077453613 network.sent‚ 1.217592e+06 disk.drbd1828.in‚0 disk.sda.in‚0 disk.drbd1552.in‚0 disk.rbd1.out‚ 0.158203125 disk.drbd2041.out‚0 disk.dm-0.in‚0 disk.drbd1938.in‚0 disk.drbd1584.in‚0 disk.sda1.out‚0 cpu‚14.272032139489431 disk.drbd1965.out‚0 disk.drbd1938.out‚0 disk.dm-0.out‚ 17.9013671875 disk.rbd0.in‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.drbd1356.in‚0 disk.drbd1552.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1285.out‚0 disk.sda2.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 network.recv‚129131# disk./.usageGB‚570.327564239502 disk.drbd1673.out‚0" disk.nvme0n1.out‚ 17.9013671875›ÞŠ0‘+ _wandbruntime‚582  _runtime‚582XÿmÄ’¾ & lr-AdamW/pg1‚3.347807024441262e-05 & lr-AdamW/pg2‚3.347807024441262e-05 trainer/global_step‚1299 # _timestamp‚1.78579390871219e+09 _runtime‚ 582.790633461 _step‚100fùp…È“Â trainer/global_step‚1299 # _timestamp‚1.78579390871219e+09 _runtime‚ 582.790633461 _step‚100 & lr-AdamW/pg1‚3.347807024441262e-05 & lr-AdamW/pg2‚3.347807024441262e-05d+þ|0”+ _wandbruntime‚582  _runtime‚582ýÛ®Õ%•­O IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22907568463584 trainer/global_step‚1299 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8640479 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1143583302656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1140782730752 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1174671051776 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23549902804480 _runtime‚ 582.790755267 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8852003 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚23464917893120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚21766785732832 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22907465621100 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22321317551104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _step‚101 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22365178 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚23464917893120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5120497 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8640479 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13734357 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚23464813424128 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚23464813424128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1143495873024 WKD‡b…ñÞeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3731506 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 % _timestamp‚1.7857939087129154e+09XKàd·'–±O WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13724701 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22321317551104 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22374836 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1143495873024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3731506 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 % _timestamp‚1.7857939087129154e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22907568463584 trainer/global_step‚1299 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8640479 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1143583302656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1140782730752 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1174671051776 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23549902804480 _runtime‚ 582.790755267 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8852003 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚23464917893120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚21766785732832 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22907465621100 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22321317551104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _step‚101 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22365178 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚23464917893120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5120497 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8640479 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13734357 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚23464813424128 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚23464813424128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚21783508865792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚23482859806720 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1175573566976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8647122 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13744894 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22382344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22338484314112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22392016 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚23567856858624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8858563 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1175706331136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8647120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5124432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1144462929408 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8858817 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1141660204086 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22382344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1144375492608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13735224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1144375492608 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22392308195328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 trainer/global_step‚1299 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8647120 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22338501353472ЄÊ"Á&™»M JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1144462929408 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8858817 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1141660204086 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22382344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1144375492608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13735224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1144375492608 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22392308195328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 trainer/global_step‚1299 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8647120 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22338501353472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1144462929408 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚23568014526464 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13735224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚22392283291648 _step‚102 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚21783508865792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚23482859806720 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1175573566976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8647122 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13744894 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22382344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22338484314112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22392016 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚23567856858624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8858563 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1175706331136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8647120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5124432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0fP$Ôl0š+ _wandbruntime‚583  _runtime‚583€'þ›ø .loss/FRt1/detection_loss‚4.196677230029221 %loss/total_loss‚76.04168428257016 trainer/global_step‚1299 _step‚103 1loss/FBt1/assignment_loss‚0.36168223107233644 .loss/FBt2/detection_loss‚6.749471456763617 epoch‚0 /loss/FRt2/assignment_loss‚5.318068129221598 1loss/SRqqt1/assignment_loss‚2.541594672203064 1loss/SRqqt2/assignment_loss‚2.297406147420406 1loss/FBt2/assignment_loss‚0.16916865956773108 .loss/FRt2/detection_loss‚4.210826897621155 1loss/SRqqt1/detection_loss‚2.8465720453219547 1loss/SRqqt2/detection_loss‚2.1557946429042842 % _timestamp‚1.7857939090827498e+09 /loss/FRt1/assignment_loss‚4.906839795859463 .loss/FBt1/detection_loss‚40.28758237458533 _runtime‚ 583.159524976MëZ­œü .loss/FRt1/detection_loss‚4.196677230029221 %loss/total_loss‚76.04168428257016 trainer/global_step‚1299 _step‚103 1loss/FBt1/assignment_loss‚0.36168223107233644 .loss/FBt2/detection_loss‚6.749471456763617 epoch‚0 /loss/FRt2/assignment_loss‚5.318068129221598 1loss/SRqqt1/assignment_loss‚2.541594672203064 1loss/SRqqt2/assignment_loss‚2.297406147420406 1loss/FBt2/assignment_loss‚0.16916865956773108 .loss/FRt2/detection_loss‚4.210826897621155 1loss/SRqqt1/detection_loss‚2.8465720453219547 1loss/SRqqt2/detection_loss‚2.1557946429042842 % _timestamp‚1.7857939090827498e+09 /loss/FRt1/assignment_loss‚4.906839795859463 .loss/FBt1/detection_loss‚40.28758237458533 _runtime‚ 583.159524976g_ ‡]œ‚(:‘ öšÄÓýòç¸ gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚124.546' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.51533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú5Ç`«ž‚(:»  öšÄÓàù²¹ disk.rbd1.out‚ 0.244140625 disk.drbd1289.out‚0 disk.sda1.out‚0 disk.drbd1510.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 ýub›disk.sda2.in‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.dm-0.in‚0 disk.drbd1226.out‚0 network.recv‚131613$ disk./.usageGB‚570.3275299072266 disk.drbd2111.in‚0 disk.sda.out‚ 16.88671875 disk.drbd1931.out‚0 disk.dm-0.out‚ 18.0732421875 disk.drbd1289.in‚0 proc.cpu.threads‚29 disk.drbd1584.out‚0 disk.sda.in‚0 disk.drbd1938.out‚0 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.nvme0n1.in‚0 disk.drbd1828.out‚0 disk.drbd2118.in‚0 disk.sda1.in‚0" disk.nvme0n1.out‚ 18.0732421875 disk.drbd1356.out‚0* proc.memory.percent‚17.171186208724976 cpu‚14.255615704251715 disk.rbd0.in‚0) proc.memory.availableMB‚ 38528.4140625 disk.drbd1550.out‚0$ proc.memory.rssMB‚11253.30859375 disk.drbd1285.out‚0 disk.drbd1965.out‚0 disk.drbd1673.out‚0 disk.drbd1285.in‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd2041.in‚0 disk.drbd1817.in‚0 disk.drbd1510.out‚0 disk.sda2.out‚ 16.88671875$ memory_percent‚41.21030569076538 disk.drbd1552.in‚0 network.sent‚ 1.262654e+06 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1931.in‚0* disk./.usagePercent‚31.895701109384078 disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.drbd1965.in‚0 disk.drbd1550.in‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd1552.out‚0¯`±œŸ‚(:‘ …›ÄÓü²‰¹ gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚113.849' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.94966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áq(aA ‚(:¶  …›ÄÓ¥‘Ϲ" disk.nvme0n1.out‚ 18.9169921875 disk.drbd1226.out‚0 network.sent‚ 1.307765e+06 network.recv‚134309 disk.drbd1965.out‚0 disk.dm-0.out‚ 18.9169921875 disk.rbd1.out‚ 0.306640625 disk.drbd1828.out‚0 disk.drbd1817.in‚0$ proc.memory.rssMB‚11253.30859375 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.sda2.in‚0 disk.sda2.out‚ 17.1015625 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd2118.out‚0 disk.drbd1510.out‚0 disk.rbd0.out‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚0 disk.drbd1356.in‚0 disk.drbd1584.out‚0* disk./.usagePercent‚31.895648628275232 disk.drbd1285.out‚0 disk.drbd1356.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 17.1015625 disk.drbd1931.in‚0 disk.drbd1550.in‚0 disk.drbd2111.in‚0 disk.drbd1584.in‚0 disk.drbd2041.in‚0 disk.drbd1938.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1938.out‚0 disk.drbd1289.out‚0 disk.drbd1931.out‚0 cpu‚14.239730848516958 disk.drbd1817.out‚0% memory_percent‚41.209983825683594% proc.memory.availableMB‚ 38528.625* proc.memory.percent‚17.171186208724976$ disk./.usageGB‚570.3265914916992 disk.rbd1.in‚0 disk.drbd2078.in‚0 disk.sda.in‚0 disk.dm-0.in‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 proc.cpu.threads‚29‚ë±0¡+ _wandbruntime‚604  _runtime‚604SIœ¦Æ¢À & lr-AdamW/pg1‚3.476667956867793e-05 & lr-AdamW/pg2‚3.476667956867793e-05 trainer/global_step‚1349 % _timestamp‚1.7857939306322584e+09 _runtime‚ 604.709215894 _step‚104€ïçŸÊ£Ä % _timestamp‚1.7857939306322584e+09 _runtime‚ 604.709215894 _step‚104 & lr-AdamW/pg1‚3.476667956867793e-05 & lr-AdamW/pg2‚3.476667956867793e-05 trainer/global_step‚1349h2Ë÷¦0¤+ _wandbruntime‚604  _runtime‚604Çâ} ³'¥­O ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1187492008448 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24367252178944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5317345 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1184671194260 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14262359 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚23179655701504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5317347 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14252703 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24367147709952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22603794348832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23235432014848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1184585391136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚23179672740864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8972629 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3874452 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24455433311744 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 % _timestamp‚1.7857939306332605e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14252703 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23235456918528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚23788362700608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _step‚105 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚23179672740864 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚23179655701504 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24367147709952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9191544 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8972627 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23234988 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1187492008448 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚24455281928192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8972627 _runtime‚ 604.713612184 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8972629 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9191799 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1219976393216 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1187579438080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632ëpÄ·'¦±O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8972629 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9191799 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1219976393216 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1187579438080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1187492008448 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24367252178944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5317345 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1184671194260 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14262359 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚23179655701504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5317347 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14252703 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24367147709952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22603794348832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23235432014848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1184585391136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚23179672740864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8972629 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3874452 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24455433311744 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 % _timestamp‚1.7857939306332605e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14252703 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23235456918528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚23788362700608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _step‚105 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚23179672740864 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚23179655701504 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24367147709952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9191544 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8972627 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23234988 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1187492008448 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚24455281928192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8972627 _runtime‚ 604.713612184 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23252168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1188371628032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8979272 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24385298568704 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚23196822464512 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22620517481792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1185548667594 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1188459064832 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23805980346150 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5321282 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23252168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1188459064832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23252664035328 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23252639131648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3877208 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDevŒŠ… iceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚23196839503872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14263226 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 605.07962808 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14263226 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1220748979712 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23242496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚23196822464512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5321284 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14272896 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9198248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121O^=éÀ&©ºM NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9198492 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1185462864358 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23242496 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚24473388111360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8979270 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚23196839503872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14263226 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 605.07962808 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14263226 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1220748979712 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23242496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚23196822464512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5321284 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14272896 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9198248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8979270 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1220885938176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚24385298568704 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8979272 _step‚106 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚24385194092544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3876966 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23252168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1188371628032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8979272 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24385298568704 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚23196822464512 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22620517481792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1185548667594 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1188459064832 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23805980346150 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5321282 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23252168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1188459064832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23252664035328 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23252639131648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3877208 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24037993851904 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚24095787854848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚25269481995776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14790361 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9530523 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1231575573504 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚23440785925472 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9304779 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1231488143872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9530796 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5514197 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9304779 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1231575573504-A¢‘3´°O JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚25360946441728 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚1399 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14780705 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚25269481995776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚23440802964832 % _timestamp‚1.7857939525933313e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24037993851904 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚24095787854848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚25269481995776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14790361 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9530523 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1231575573504 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚23440785925472 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9304779 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1231488143872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9530796 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5514197 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9304779 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1231575573504 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1231488143872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0m_ª ,0µ+ _wandbruntime‚627  _runtime‚627ïaܽ&¶·M YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚24055160614912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1266034757632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24112320 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24055177654272 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚25287528378368 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14791228 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 trainer/global_step‚1399 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5518132 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 _step‚110 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚25287632854528 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚24112994971648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9537270 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1232367763456 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚24055177654272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 _runtime‚ 627.041128546 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24112320 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9311422 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1232455200256 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7857939529637616e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4019138 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1232455200256 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1265899896320 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23457543137152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25378894867968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1229351327866 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1229437131102 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4019396 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚24686980268254 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚24686877425658 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14800898 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚24055160614912 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚25287632854528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24102648 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚23457526097792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14800898 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9311420 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121þMŲÁ&·»M S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚25287632854528 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚24112994971648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9537270 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1232367763456 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚24055177654272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 _runtime‚ 627.041128546 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24112320 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9311422 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1232455200256 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7857939529637616e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4019138 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1232455200256 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1265899896320 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23457543137152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25378894867968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1229351327866 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1229437131102 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4019396 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚24686980268254 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚24686877425658 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14800898 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚24055160614912 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚25287632854528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24102648 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚23457526097792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14800898 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9311420 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚24055160614912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1266034757632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24112320 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24055177654272 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚25287528378368 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14791228 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 trainer/global_step‚1399 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5518132 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 _step‚110 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887nEÝÊÆ0¸+ _wandbruntime‚627  _runtime‚627w[¹ü epoch‚0 1loss/FBt1/assignment_loss‚0.30325934402644633 1loss/SRqqt1/detection_loss‚2.9467819945225395 % _timestamp‚1.7857939529655397e+09 _runtime‚ 627.042307163 2loss/SRqqt2/assignment_loss‚2.2138123641142973 1loss/FBt2/assignment_loss‚0.10407405189616839 /loss/FRt1/detection_loss‚3.4904887027293445 .loss/FRt2/detection_loss‚5.150694146508076 %loss/total_loss‚64.68026866232009 _step‚111 /loss/FRt1/assignment_loss‚4.584188027381897 /loss/FRt2/assignment_loss‚5.031708588365649 1loss/SRqqt2/detection_loss‚2.5123452173779013 /loss/FBt1/detection_loss‚30.730817271712294 trainer/global_step‚1399 2loss/SRqqt1/assignment_loss‚2.4443505354798756 .loss/FBt2/detection_loss‚5.167748418205613ƒ¬éº€ 2loss/SRqqt1/assignment_loss‚2.4443505354798756 .loss/FBt2/detection_loss‚5.167748418205613 epoch‚0 1loss/FBt1/assignment_loss‚0.30325934402644633 1loss/SRqqt1/detection_loss‚2.9467819945225395 % _timestamp‚1.7857939529655397e+09 _runtime‚ 627.042307163 2loss/SRqqt2/assignment_loss‚2.2138123641142973 1loss/FBt2/assignment_loss‚0.10407405189616839 /loss/FRt1/detection_loss‚3.4904887027293445 .loss/FRt2/detection_loss‚5.150694146508076 %loss/total_loss‚64.68026866232009 _step‚111 /loss/FRt1/assignment_loss‚4.584188027381897 /loss/FRt2/assignment_loss‚5.031708588365649 1loss/SRqqt2/detection_loss‚2.5123452173779013 /loss/FBt1/detection_loss‚30.730817271712294 trainer/global_step‚1399oÊ$5†‘»‚(:† £›ÄÓ¦ø†¹ gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚130.368' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚43.456 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0";B¼‚(:·  £›ÄÓØÚÓ¹ disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1584.in‚0* proc.memory.percent‚17.171186208724976 cpu‚14.210927615211776 disk.drbd1584.out‚0 disk.drbd2118.in‚0$ proc.memory.rssMB‚11253.30859375 disk.sda1.out‚0 disk.drbd1226.out‚0 disk.rbd1.in‚0 disk.drbd1828.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.sda2.out‚ 17.76171875 network.sent‚ 1.35625e+06$ disk./.usageGB‚570.3275947570801 disk.drbd1550.out‚0 disk.drbd1931.in‚0( proc.memory.availableMB‚ 38528.296875 proc.cpu.threads‚29 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1510.out‚0 disk.drbd1285.out‚0 disk.rbd1.out‚ 0.3388671875 network.recv‚138648 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1289.out‚0 disk.sda.in‚0 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 17.76171875 disk.sda1.in‚0 disk.drbd1510.in‚0 disk.drbd1285.in‚0 disk.drbd1356.out‚0 disk.rbd0.in‚0 disk.drbd2078.in‚0 disk.drbd1673.in‚0 disk.sda2.in‚0$ memory_percent‚41.21048450469971 disk.drbd2078.out‚0) disk./.usagePercent‚31.89570473612737 disk.drbd1828.out‚0 disk.dm-0.in‚0 disk.dm-0.out‚ 19.810546875 disk.drbd1552.in‚0 disk.drbd1550.in‚0 disk.nvme0n1.in‚0! disk.nvme0n1.out‚ 19.810546875k†6o‘½‚(:† ²›ÄÓçݸ gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚120.447' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.149 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹_>¾‚(:³  ²›ÄÓÛ•ž¹ disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.dm-0.out‚ 19.994140625 disk.nvme0n1.in‚0 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1356.in‚0 disk.drbd1289.in‚0 cpu‚14.19754255944381 disk.drbd1584.out‚0 disk.drbd2041.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.sda2.in‚0 proc.cpu.threads‚29 disk.sda1.in‚0 disk.drbd1356.out‚0 disk.drbd1289.out‚0 proc.memory.rssMB‚ 11253.3125 disk.drbd1285.in‚0 disk.rbd1.out‚ 0.3388671875 network.recv‚141477$ disk./.usageGB‚570.3275947570801 disk.drbd2041.out‚0 network.sent‚ 1.401436e+06 disk.drbd1226.in‚0* proc.memory.percent‚17.171192169189453 disk.drbd2111.in‚0 disk.drbd2078.out‚0 disk.sda.out‚ 18.02734375 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.drbd1931.in‚0 disk.drbd1817.out‚0 disk.drbd1285.out‚0 disk.sda2.out‚ 18.02734375 disk.drbd2078.in‚0 disk.dm-0.in‚0 disk.drbd1817.in‚0 disk.drbd1965.out‚0) disk./.usagePercent‚31.89570473612737 disk.drbd2118.out‚0 disk.sda.in‚0 disk.drbd1510.out‚0 disk.rbd0.in‚0 disk.drbd1550.out‚0( proc.memory.availableMB‚ 38528.046875 disk.drbd1673.in‚0 disk.drbd2118.in‚0 disk.rbd0.out‚0$ memory_percent‚41.21086597442627 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1673.out‚0 disk.drbd1510.in‚0! disk.nvme0n1.out‚ 19.994140625 disk.drbd1226.out‚0 disk.rbd1.in‚03Z0¿+ _wandbruntime‚648  _runtime‚648[]ýÛÈÀ trainer/global_step‚1449 % _timestamp‚1.7857939745423493e+09 _runtime‚ 648.619296905 _step‚112 ' lr-AdamW/pg1‚3.7343898217208536e-05 ' lr-AdamW/pg2‚3.7343898217208536e-05‹êDàÌÁÆ % _timestamp‚1.7857939745423493e+09 _runtime‚ 648.619296905 _step‚112 ' lr-AdamW/pg1‚3.7343898217208536e-05 ' lr-AdamW/pg2‚3.7343898217208536e-05 trainer/global_step‚1449pBr³0Â+ _wandbruntime‚648  _runtime‚648çê‘ Ã­O L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15308707 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24945634 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24955292 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1272362318152 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1275571708928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 % _timestamp‚1.7857939745433726e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9636929 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9636927 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1310284670464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26171816281600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚24277811580832 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9870194 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1275571708928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1275484279296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚26266295593984 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26171816281600 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9636929 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚24956143694848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24896349041664 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9636927 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15318363 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9869933 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24896332002304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚ 648.620575417 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 trainer/global_step‚1449 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1275484279296 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5711047 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1272448121276 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚24277794541472 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚26171920750592 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚26171920750592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4158888 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24956168598528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1310151899136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0MÚ›ï·'ıO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚26266295593984 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26171816281600 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9636929 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚24956143694848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24896349041664 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9636927 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15318363 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9869933 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24896332002304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚ 648.620575417 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 trainer/global_step‚1449 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1275484279296 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5711047 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1272448121276 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚24277794541472 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚26171920750592 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚26171920750592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4158888 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24956168598528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1310151899136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15308707 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24945634 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24955292 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1272362318152 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1275571708928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 % _timestamp‚1.7857939745433726e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9636929 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9636927 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1310284670464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26171816281600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚24277811580832 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9870194 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1275571708928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1275484279296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0q8D­Ê0Å+ _wandbruntime‚648  _runtime‚648v1d©¼&ƶM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚26189862664192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9643570 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚26189967140352 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚24913498765312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1276363898880 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24913515804672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5714984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1273239791374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4161915 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1276451335680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 trainer/global_step‚1449 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15319230 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26189967140352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26284560190464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9643572 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24962800 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚24973375715328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚24913515804672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1311184475136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1276451335680 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24972472 % _timestamp‚1.7857939749120538e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24972472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5714982 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9876664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1276363898880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚24294534713792 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15319230 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1273325594610 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15328900 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4161682 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚24294551753152 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _step‚114 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24962800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9643572 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚25567877347762 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360X©î¯ÇºM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24972472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5714982 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9876664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1276363898880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚24294534713792 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15319230 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1273325594610 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15328900 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4161682 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚24294551753152 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _step‚114 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24962800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9643572 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚25567877347762 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚26189862664192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_byt6Ô,G¨es.small_pool.peak‚ 145651200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9643570 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚26189967140352 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚24913498765312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1276363898880 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24913515804672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5714984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1273239791374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4161915 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1276451335680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 trainer/global_step‚1449 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15319230 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26189967140352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26284560190464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9643572 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24962800 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚24973375715328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚24913515804672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1311184475136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1276451335680 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24972472 % _timestamp‚1.7857939749120538e+09r"3k 0È+ _wandbruntime‚648  _runtime‚648Rä¹´ÿÉù .loss/FBt2/detection_loss‚5.357717567312112 %loss/total_loss‚70.55247031019604 0loss/FRt2/assignment_loss‚5.1645735221750595 2loss/SRqqt2/assignment_loss‚2.2748753247636087 0loss/FBt2/assignment_loss‚0.5418965052871499 /loss/FRt1/detection_loss‚3.4044337859925102 .loss/FRt2/detection_loss‚4.653774237808059 _step‚115 _runtime‚ 648.991364084 epoch‚0 /loss/FRt1/assignment_loss‚4.287754601123286 1loss/FBt1/assignment_loss‚0.16519127873471007 .loss/FBt1/detection_loss‚38.36055344272289 trainer/global_step‚1449 2loss/SRqqt1/assignment_loss‚1.9775173510114352 1loss/SRqqt1/detection_loss‚2.1243043239968906 0loss/SRqqt2/detection_loss‚2.239878369268325 $ _timestamp‚1.785793974914526e+09 þÊý /loss/FRt1/detection_loss‚3.4044337859925102 .loss/FRt2/detection_loss‚4.653774237808059 _step‚115 _runtime‚ 648.991364084 epoch‚0 /loss/FRt1/assignment_loss‚4.287754601123286 1loss/FBt1/assignment_loss‚0.16519127873471007 .loss/FBt1/detection_loss‚38.36055344272289 trainer/global_step‚1449 2loss/SRqqt1/assignment_loss‚1.9775173510114352 1loss/SRqqt1/detection_loss‚2.1243043239968906 0loss/SRqqt2/detection_loss‚2.239878369268325 $ _timestamp‚1.785793974914526e+09 .loss/FBt2/detection_loss‚5.357717567312112 %loss/total_loss‚70.55247031019604 0loss/FRt2/assignment_loss‚5.1645735221750595 2loss/SRqqt2/assignment_loss‚2.2748753247636087 0loss/FBt2/assignment_loss‚0.5418965052871499s¹"Ĝ˂(:‘ Á›ÄÓ웹 gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚121.904' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.63466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zFæ½BÌ‚(:·  Á›ÄÓ°Åá¹ disk.drbd1285.out‚0 disk.drbd2078.out‚0 proc.cpu.threads‚29 disk.sda1.in‚0* proc.memory.percent‚17.171192169189453 disk.sda2.out‚ 18.41015625) disk./.usagePercent‚31.89570473612737 disk.drbd1356.in‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.sda1.out‚0 disk.sda.out‚ 18.41015625 disk.dm-0.in‚0 disk.drbd1673.in‚0 disk.nvme0n1.in‚0 disk.drbd1828.out‚0$ disk./.usageGB‚570.3275947570801 disk.drbd1550.in‚0 disk.drbd2111.out‚0 disk.drbd1965.out‚0 disk.rbd1.out‚ 0.40185546875 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd1931.out‚0 disk.dm-0.out‚ 20.3896484375 proc.memory.rssMB‚ 11253.3125 disk.drbd1673.out‚0 disk.drbd1226.in‚0 disk.drbd1285.in‚0 disk.rbd0.out‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd2041.out‚0 disk.drbd1510.in‚0 network.sent‚ 1.40456e+06 disk.drbd1550.out‚0 disk.drbd1828.in‚0" disk.nvme0n1.out‚ 20.3896484375 disk.drbd1552.out‚0 disk.drbd1289.in‚0 disk.sda.in‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0* proc.memory.availableMB‚38528.22265625 disk.drbd1552.in‚0$ memory_percent‚41.21059775352478 disk.rbd0.in‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 cpu‚14.19344371212544 network.recv‚142842 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.drbd1584.out‚0 disk.drbd1938.out‚0 disk.drbd1510.out‚0,˜y´0Í+ _wandbruntime‚670  _runtime‚670LvŒïÅο & lr-AdamW/pg1‚3.863250754147384e-05 & lr-AdamW/pg2‚3.863250754147384e-05 trainer/global_step‚1499 % _timestamp‚1.7857939964014454e+09 _runtime‚ 670.47841905 _step‚116šötBÉÏà _step‚116 & lr-AdamW/pg1‚3.863250754147384e-05 & lr-AdamW/pg2‚3.863250754147384e-05 trainer/global_step‚1499 % _timestamp‚1.7857939964014454e+09 _runtime‚ 670.47841905t#Fš0Ð+ _wandbruntime‚670  _runtime‚670 s—³'Ñ­O RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9969077 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122285568 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚26431156781616 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚27074255036416 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15836709 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10209248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4301605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27172070955008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25114820196832 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚25754687192064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1319480414720 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25816499534848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1319567844352 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚25754670152704 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27074150567424 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚25114803157472 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15846365 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5907897 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1499 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27171923765760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1355546516480 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9969079 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1316250781660 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 _step‚117 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25815444 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15846365 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4301353 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1316336584784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1319567844352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1319480414720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1355424230912 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15836709 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 # _timestamp‚1.78579399640248e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25816524438528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚27074150567424 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10209502 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9969079 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147189248 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25754670152704 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27074150567424 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚25114803157472 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15846365 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5907897 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1499 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27171923765760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1355546516480 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9969079 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1316250781660 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 _step‚117 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25815444 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15846365 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4301353 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1316336584784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1319567844352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1319480414720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1355424230912 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15836709 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 # _timestamp‚1.78579399640248e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25816524438528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚27074150567424 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10209502 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9969079 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147189248 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25754670152704 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15856902 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5911834 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚25771836915712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27190026647552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9975720 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _step‚118 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9975722 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4304463 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1317128254882 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1317214058118 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1320447471104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27190190606848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚25131543329792 trainer/global_step‚1499 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15847232 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚26448774427270 _runtime‚ 670.847891007 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9975722 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚25131560369152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9975720 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚25771853955072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚25833706651648 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 % _timestamp‚1.7857939967706094e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1320360034304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚27092196950016 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25832624 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15847232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1320360034304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1356459051520 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25832624 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26448671584674 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚25771853955072 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4304198 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1320447471104 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚25771836915712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10216030 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672~Ïý²Ã&Õ½M GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚25131560369152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9975720 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚25771853955072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚25833706651648 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 % _timestamp‚1.7857939967706094e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1320360034304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚27092196950016 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25832624 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15847232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1320360034304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1356459051520 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25832624 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26448671584674 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚25771853955072 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4304198 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1320447471104 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚25771836915712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10216030 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15856902 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5911834 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚25771836915712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27190026647552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9975720 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _step‚118 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9975722 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4304463 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1317128254882 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1317214058118 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1320447471104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27190190606848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚25131543329792 trainer/global_step‚1499 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15847232 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚26448774427270 _runtime‚ 670.847891007 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9975722 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28077597832704 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚25951811773472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10301229 _step‚121 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16374367 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16364711 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4444759 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1400717554176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚1549 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1363476550144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26676855374848 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚26613025342464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚26613008303104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16364711 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4445011 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26676880278528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚27976589322240 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27976484853248 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26675596 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26665938 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1363563979776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16374367 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25951828812832 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10301227 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚26613025342464 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚26613008303104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7857940189155672e+09 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28077438060544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚27976484853248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10549504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27312053861124 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1360139245168 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚27976589322240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1363563979776 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10301229 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1363476550144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6104747 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27311951018640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10301227 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1210¯3—Ûâ·O G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 692.992842347 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10549758 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.curà½üâ$rent‚87429632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28077597832704 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚25951811773472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10301229 _step‚121 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16374367 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16364711 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4444759 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1400717554176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚1549 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1363476550144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26676855374848 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚26613025342464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚26613008303104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16364711 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4445011 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26676880278528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚27976589322240 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27976484853248 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26675596 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26665938 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1363563979776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16374367 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25951828812832 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10301227 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚26613025342464 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚26613008303104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7857940189155672e+09 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28077438060544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚27976484853248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10549504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27312053861124 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1360139245168 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚27976589322240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1363563979776 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10301229 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1363476550144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6104747 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27311951018640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10301227 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0yI[0ã+ _wandbruntime‚693  _runtime‚693YëÃ&ä½M C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26692776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚26630192105472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27329568664182 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚27994635712000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26694087395328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10556477 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 % _timestamp‚1.7857940192945356e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16375234 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27329671506778 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10307870 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16384904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _step‚122 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚27994635712000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6108682 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28095546308096 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26692776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _runtime‚ 693.371891019 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16375234 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1364443606528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚27994531235840 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚26630175066112 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26694062491648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28095706073088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1361016718390 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10307872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1401618677760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 trainer/global_step‚1549 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚25968568985152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26630192105472 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1364356169728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1364443606528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚25968551945792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚26630175066112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1364356169728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1361102521626 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10556207 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10307870 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 trainer/global_step‚1549 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚25968568985152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26630192105472 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1364356169728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1364443606528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚25968551945792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚26630175066112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1364356169728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1361102521626 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10556207 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10307870 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 % _timestamp‚1.7857940192945356e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16375234 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27329671506778 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10307870 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16384904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _step‚122 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚27994635712000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6108682 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28095546308096 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26692776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _runtime‚ 693.371891019 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16375234 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1364443606528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚27994531235840 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚26630175066112 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26694062491648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28095706073088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0zù1· 0æ+ _wandbruntime‚693  _runtime‚693NØ&¥üçö 2loss/SRqqt1/assignment_loss‚1.6934626398438757 0loss/SRqqt1/detection_loss‚2.177568983290811 0loss/SRqqt2/detection_loss‚2.630535571574809 epoch‚0 /loss/FRt2/assignment_loss‚4.952012100219727 .loss/FRt1/detection_loss‚3.832085566719373 $loss/total_loss‚51.7944075112708 $ _timestamp‚1.785794019296367e+09 0loss/FRt1/assignment_loss‚3.9836606634987723 1loss/SRqqt2/assignment_loss‚2.442095720904997 0loss/FBt1/assignment_loss‚0.1445760690034009 1loss/FBt2/assignment_loss‚0.09709535001976681 trainer/global_step‚1549 _runtime‚ 693.373152478 .loss/FRt2/detection_loss‚6.155936059951782 .loss/FBt1/detection_loss‚20.74173341180407 /loss/FBt2/detection_loss‚2.9436453744394107 _step‚123Tçãšèú $loss/total_loss‚51.7944075112708 $ _timestamp‚1.785794019296367e+09 0loss/FRt1/assignment_loss‚3.9836606634987723 1loss/SRqqt2/assignment_loss‚2.442095720904997 0loss/FBt1/assignment_loss‚0.1445760690034009 1loss/FBt2/assignment_loss‚0.09709535001976681 trainer/global_step‚1549 _runtime‚ 693.373152478 .loss/FRt2/detection_loss‚6.155936059951782 .loss/FBt1/detection_loss‚20.74173341180407 /loss/FBt2/detection_loss‚2.9436453744394107 _step‚123 2loss/SRqqt1/assignment_loss‚1.6934626398438757 0loss/SRqqt1/detection_loss‚2.177568983290811 0loss/SRqqt2/detection_loss‚2.630535571574809 epoch‚0 /loss/FRt2/assignment_loss‚4.952012100219727 .loss/FRt1/detection_loss‚3.832085566719373{ˆä R‘é‚(:† î›ÄÓùÄû¸ gpu.0.gpu‚36.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚121.674' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.558 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡U˜øhê‚(:±  î›ÄÓٞȹ network.recv‚148696 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd1510.in‚0 disk.drbd1673.in‚0 disk.drbd1817.out‚0# disk.nvme0n1.out‚21.65185546875 disk.drbd2111.out‚0 disk.sda.in‚0$ disk./.usageGB‚570.3276023864746 disk.rbd1.out‚ 0.46484375 disk.drbd1289.out‚0 disk.drbd1550.in‚0 disk.dm-0.out‚21.65185546875$ memory_percent‚41.21084213256836 disk.nvme0n1.in‚0 disk.rbd0.out‚0 disk.drbd1828.out‚0 disk.sda.out‚ 19.390625 disk.drbd1673.out‚0 disk.drbd1285.out‚0 disk.sda2.out‚ 19.390625 disk.drbd2041.out‚0 disk.drbd1931.in‚0 proc.cpu.threads‚29 network.sent‚ 1.497336e+06 disk.sda2.in‚0 disk.drbd2078.out‚0 disk.drbd2118.out‚0 disk.sda1.in‚0 disk.drbd1226.out‚0 disk.drbd1965.in‚0* proc.memory.percent‚17.171192169189453 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.drbd2111.in‚0 disk.drbd2118.in‚0 disk.drbd1356.out‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 cpu‚14.187030597452623 disk.drbd1356.in‚0 disk.drbd1965.out‚0 disk.sda1.out‚0 disk.drbd1938.out‚0 disk.dm-0.in‚0& proc.memory.availableMB‚ 38528.0625 disk.drbd1510.out‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd1289.in‚0 disk.drbd1584.in‚0 disk.dÜ$¼Ôrbd1226.in‚0* disk./.usagePercent‚31.895705162803058 disk.drbd1552.in‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd2041.in‚0 proc.memory.rssMB‚ 11253.3125 disk.drbd1938.in‚0Ö]ÔX0ë+ _wandbruntime‚715  _runtime‚715`ávoÆìÀ % _timestamp‚1.7857940411833897e+09 _runtime‚ 715.260361077 _step‚124 & lr-AdamW/pg1‚4.120972619000445e-05 & lr-AdamW/pg2‚4.120972619000445e-05 trainer/global_step‚1599Ã@Õ‰ÊíÄ _step‚124 & lr-AdamW/pg1‚4.120972619000445e-05 & lr-AdamW/pg2‚4.120972619000445e-05 trainer/global_step‚1599 % _timestamp‚1.7857940411833897e+09 _runtime‚ 715.260361077|f}8O0î+ _wandbruntime‚715  _runtime‚715É /ú¹'ï³O RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27537211214848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4588649 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1407560115200 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6301597 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10633377 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16902369 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10890002 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28983143298048 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1445907179520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28878923608064 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28878819139072 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4588407 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚26788837428832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10633379 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10633377 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27471363492864 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28982985623040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚28192848098148 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1407472685568 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16902369 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27471346453504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1404113511800 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27471346453504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1404027708676 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10890246 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27537236118528 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28878923608064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1445774408192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27471363492864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6301595 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 % _timestamp‚1.7857940411844592e+09 _runtime‚ 715.261628812 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1407472685568{£¡½'ð·O [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27471346453504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1404113511800 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27471346453504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1404027708676 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10890246 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27537236118528 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28878923608064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1445774408192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27471363492864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6301595 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 % _timestamp‚1.7857940411844592e+09 _runtime‚ 715.261628812 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1407472685568 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27537211214848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4588649 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1407560115200 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6301597 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10633377 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16902369 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10890002 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28983143298048 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1445907179520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28878923608064 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28878819139072 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4588407 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚26788837428832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10633379 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10633377 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27471363492864 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28982985623040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚28192848098148 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1407472685568 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16902369}£µ)V0ñ+ _wandbruntime‚715  _runtime‚715ÆÓ)Ã&ò½M UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4591504 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29001253144064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1446809908736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10640020 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10896792 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1599 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1408352305152 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28896865521664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6305534 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27552928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28896969997824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1408352305152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4591260 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1446679241728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27543256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27554418331648 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16912906 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚27488513216512 % _timestamp‚1.7857940415530167e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28210568586286 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27543256 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10640022 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27488530255872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27554443235328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚26805577601152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚27488530255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚ 715.630389285 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1404990985134 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10640022 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚26805560561792 _step‚126 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0+äìÏóÁM >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27554443235328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚26805577601152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚27488530255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _h<,A: runtime‚ 715.630389285 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1404990985134 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10640022 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚26805560561792 _step‚126 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4591504 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29001253144064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1446809908736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10640020 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10896792 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1599 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1408352305152 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28896865521664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6305534 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27552928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28896969997824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1408352305152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4591260 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1446679241728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27543256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27554418331648 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16912906 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚27488513216512 % _timestamp‚1.7857940415530167e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28210568586286 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27543256 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10640022 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27488530255872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17430371 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚28329701643264 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1451468820992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10965527 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚28329684603904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10965529 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11230251 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6498447 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10965527 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28329684603904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚29888492700672 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28395900 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚1649 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28397567054848 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚29781153424896 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1451556250624 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4731551 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1490925645824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17430371Y×}Ù€¸O 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚129 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28386242 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10965529 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚29888644084224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6498445 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28397591958528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 % _timestamp‚1.7857940633669996e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17430371 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚28329701643264 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1451468820992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10965527 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚28329684603904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10965529 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11230251 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6498447 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10965527 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28329684603904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚29888492700672 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28395900 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚1649 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28397567054848 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚29781153424896 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1451556250624 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4731551 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1490ß1¡Üå925645824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17430371 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1451556250624 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17420715 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11229996 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4731804 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1447916172184 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 _runtime‚ 737.444210226 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚29781257893888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1451468820992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1491052125696 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887®5ÉS0+ _wandbruntime‚737  _runtime‚737¶~^ÑÃ&‚½M J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17431238 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27642569177792 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17440908 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚29799304283648 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17431238 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28414799075328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 % _timestamp‚1.7857940637383657e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚29799199807488 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11236714 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28403408 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28346851366912 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6502384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚1649 _runtime‚ 737.815701957 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1452348440576 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28346868406272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚29091465665794 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10972172 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6502382 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28414774171648 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29091362823198 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28403408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1491961990144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28413080 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27642586217152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1448879448642 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28413080 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1452435877376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28346851366912 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17440908 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29906761065472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11236951 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0IƒÈ&ƒÂM R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚29799199807488 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11236714 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28403408 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28346851366912 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6502384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚1649 _runtime‚ 737.815701957 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1452348440576 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28346868406272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚29091465665794 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10972172 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6502382 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28414774171648 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29091362823198 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28403408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1491961990144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28413080 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27642586217152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1448879448642 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28413080 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1452435877376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28346851366912 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17440908 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29906761065472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11236951 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17431238 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27642569177792 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17440908 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚29799304283648 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17431238 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28414799075328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 % _timestamp‚1.7857940637383657e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1‚%D0„+ _wandbruntime‚737  _runtime‚737ƒ™t®…û /loss/FRt2/assignment_loss‚4.879835527283805 2loss/SRqqt1/assignment_loss‚1.7863415622357595 /loss/FBt1/detection_loss‚48.743099922935166 .loss/FBt2/detection_loss‚4.637465124439359 % _timestamp‚1.7857940637401123e+09 0loss/FRt1/assignment_loss‚3.9373013106259434 2loss/SRqqt2/assignment_loss‚2.0157608848878708 1loss/FBt1/assignment_loss‚0.17559713132989904 .loss/FRt1/detection_loss‚3.031138071011413 epoch‚0 _runtime‚ 737.816887128 1loss/FBt2/assignment_loss‚0.11049613808669771 .loss/FRt2/detection_loss‚4.398444028411593 1loss/SRqqt1/detection_loss‚2.1816368503166474 0loss/SRqqt2/detection_loss‚2.444072749914955 %loss/total_loss‚78.34118930147912 trainer/global_step‚1649 _step‚131ÁÚkц€ .loss/FBt2/detection_loss‚4.637465124439359 % _timestamp‚1.7857940637401123e+09 0loss/FRt1/assignment_loss‚3.9373013106259434 2loss/SRqqt2/assignment_loss‚2.0157608848878708 1loss/FBt1/assignment_loss‚0.17559713132989904 .loss/FRt1/detection_loss‚3.031138071011413 epoch‚0 _runtime‚ 737.816887128 1loss/FBt2/assignment_loss‚0.11049613808669771 .loss/FRt2/detection_loss‚4.398444028411593 1loss/SRqqt1/detection_loss‚2.1816368503166474 0loss/SRqqt2/detection_loss‚2.444072749914955 %loss/total_loss‚78.34118930147912 trainer/global_step‚1649 _step‚131 /loss/FRt2/assignment_loss‚4.879835527283805 2loss/SRqqt1/assignment_loss‚1.7863415622357595 /loss/FBt1/detection_loss‚48.743099922935166ƒÛïXßœ‡‚(:‘ ›œÄӕ¹ gpu.0.gpu‚9.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚122.183' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.727666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Jm¯zEˆ‚(:º  ›œÄÓÌØϹ disk.drbd1817.in‚0 disk.sda.in‚0 disk.sda.out‚ 20.33203125 disk.dm-0.in‚0$ disk./.usageGB‚570.3276786804199 disk.nvme0n1.in‚0 disk.drbd1552.out‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd1552.in‚0 disk.rbd1.out‚ 0.46484375 disk.drbd1550.out‚0* proc.memory.percent‚17.171192169189453 disk.drbd2118.out‚0 disk.rbd1.in‚0 disk.drbd1584.in‚0# disk.nvme0n1.out‚23.25244140625 disk.drbd2041.in‚0 disk.drbd1356.out‚0 cpu‚14.166928222462221 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd1285.in‚0 disk.drbd1965.out‚0 disk.drbd1289.in‚0 proc.cpu.threads‚29 disk.drbd2118.in‚0 disk.dm-0.out‚23.25244140625 disk.drbd1584.out‚0 disk.drbd2111.out‚0 disk.drbd1931.in‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.sda2.in‚0% memory_percent‚41.211336851119995 disk.drbd1931.out‚0 disk.drbd1673.out‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd1828.out‚0 proc.memory.rssMB‚ 11253.3125* disk./.usagePercent‚31.895709429559872 disk.drbd1817.out‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1938.in‚0 disk.drbd1510.out‚0 network.sent‚ 1.590366e+06 disk.sda1.out‚0 disk.drbd1356.in‚0 disk.rbd0.out‚0 network.recv‚154456 disk.sda1.in‚0 disk.sda2.out‚ 20.33203125 disk.drbd1938.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0* proc.memory.availableMB‚38527.73828125 disk.rbd0.in‚0 disk.drbd2111.in‚0Æc_0‰+ _wandbruntime‚759  _runtime‚759ûGoÆŠÀ % _timestamp‚1.7857940858093753e+09 _runtime‚ 759.886326679 _step‚132 & lr-AdamW/pg1‚4.378694483853506e-05 & lr-AdamW/pg2‚4.378694483853506e-05 trainer/global_step‚16994)÷ÅË‹Å trainer/global_step‚1699 % _timestamp‚1.7857940858093753e+09 _runtime‚ 759.886326679 _step‚132 & lr-AdamW/pg1‚4.378694483853506e-05 & lr-AdamW/pg2‚4.378694483853506e-05„vC³0Œ+ _wandbruntime‚759  _runtime‚759nÿ™Ÿõ³O QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1491804635692 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1495552386048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1536114652672 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1495464956416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29188039793664 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30683487710720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30683592179712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11569380 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 _runtime‚ 759.887640167 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11297679 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17958373 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29256052 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚28462837621472 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29246394 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17948717 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1536245326848 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11297679 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1491890438816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29188039793664 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17948717 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30683487710720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1495552386048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11297677 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30683592179712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚29188022754304 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17958373 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚29257922894848 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29954642257164 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚28462854660832 % _timestamp‚1.7857940858104258e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29188022754304 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11297677 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚30794193125376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚1699 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚30794037547520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0]Ù@¾'Ž¸O L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11297677 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚30794193125376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚1699 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚30794037547520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1491804635692 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1495552386048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1536114652672 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1495464956416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29188039793664 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30683487710720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30683592179712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11569380 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 _runtime‚ 759.887640167 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11297679 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17958373 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29256052 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚28462837621472 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29246394 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17948717 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1536245326848 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11297679 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1491890438816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29188039793664 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17948717 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30683487710720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1495552386048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11297677 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30683592179712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚29188022754304 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17958373 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚29257922894848 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29954642257164 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚28462854660832 % _timestamp‚1.7857940858104258e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29188022754304 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0…©e#€0+ _wandbruntime‚760  _runtime‚760¹ÔúBÃ&½M EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚29275130011648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11576185 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 trainer/global_step‚1699 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1492682108914 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6699234 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1496344576000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11304320 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17959240 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚29205206556672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11304322 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1496432012800 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17968910 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29273232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29273232 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6699232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30701638569472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4876951 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚29275154915328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1537028440064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29205206556672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11575939 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚30812158451712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4876707 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _step‚134 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 _runtime‚ 760.264025442 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11304322 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7857940861866217e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚30701638569472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17959240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1496344576000 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30701534093312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29263560 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1492767912150 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11304320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29273232 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6699232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30701638569472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4876951 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚29275154915328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1537028440064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29205206556672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11575939 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚30812158451712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4876707 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _step‚134 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 _runtime‚ 760.264025442 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11304322 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7857940861866217e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚30701638569472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17959240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1496344576000 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30701534093312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29263560 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1492767912150 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11304320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚29275130011648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11576185 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 trainer/global_step‚1699 L>Deviceêsõ“`StatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1492682108914 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6699234 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1496344576000†¦»ÀK0’+ _wandbruntime‚760  _runtime‚760Ußcý“÷ 0loss/FBt1/assignment_loss‚0.6206671897962224 /loss/FRt2/detection_loss‚5.4409513393683095 -loss/FBt2/detection_loss‚4.95098491649329 % _timestamp‚1.7857940861885667e+09 _step‚135 1loss/SRqqt2/assignment_loss‚2.079686634566473 1loss/SRqqt1/detection_loss‚1.9779894795030808 &loss/total_loss‚50.716585207372944 /loss/FRt1/assignment_loss‚3.704431067715894 1loss/FBt2/assignment_loss‚0.15412550217674667 .loss/FRt1/detection_loss‚3.064025725196074 0loss/SRqqt2/detection_loss‚2.494739033194531 epoch‚0 _runtime‚ 760.26535923 /loss/FRt2/assignment_loss‚4.738953066723687 1loss/SRqqt1/assignment_loss‚1.520232076780135 /loss/FBt1/detection_loss‚19.969799175858498 trainer/global_step‚1699þÈ”ü 1loss/SRqqt2/assignment_loss‚2.079686634566473 1loss/SRqqt1/detection_loss‚1.9779894795030808 &loss/total_loss‚50.716585207372944 /loss/FRt1/assignment_loss‚3.704431067715894 1loss/FBt2/assignment_loss‚0.15412550217674667 .loss/FRt1/detection_loss‚3.064025725196074 0loss/SRqqt2/detection_loss‚2.494739033194531 epoch‚0 _runtime‚ 760.26535923 /loss/FRt2/assignment_loss‚4.738953066723687 1loss/SRqqt1/assignment_loss‚1.520232076780135 /loss/FBt1/detection_loss‚19.969799175858498 trainer/global_step‚1699 0loss/FBt1/assignment_loss‚0.6206671897962224 /loss/FRt2/detection_loss‚5.4409513393683095 -loss/FBt2/detection_loss‚4.95098491649329 % _timestamp‚1.7857940861885667e+09 _step‚135‡ù«ãœ•‚(:‘ ªœÄÓ·ê㸠gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚101.068' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚33.68933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y$è‡6–‚(:«  ªœÄÓ·›¯¹ disk.drbd2078.in‚0 disk.drbd1817.out‚0 disk.nvme0n1.out‚ 23.70703125 disk.drbd1938.out‚0 disk.drbd1673.out‚0 disk.drbd1552.in‚0* proc.memory.percent‚17.171192169189453 disk.sda1.out‚0 disk.drbd1226.out‚0 disk.drbd1285.in‚0 disk.rbd0.out‚0% memory_percent‚41.211843490600586 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.sda2.in‚0 disk.drbd1285.out‚0 disk.drbd1356.out‚0* disk./.usagePercent‚31.895707296181463 disk.drbd2078.out‚0 disk.drbd1673.in‚0 disk.drbd1817.in‚0 disk.nvme0n1.in‚0 disk.drbd1931.in‚0 disk.drbd1584.out‚0 disk.drbd2041.out‚0 disk.drbd1289.out‚0 disk.dm-0.out‚ 23.70703125 proc.cpu.threads‚29 disk.drbd1289.in‚0 disk.drbd1931.out‚0 disk.drbd1510.in‚0 disk.drbd1356.in‚0$ disk./.usageGB‚570.3276405334473 disk.drbd2118.out‚0 cpu‚14.156863196644322 network.sent‚ 1.635272e+06 disk.drbd1550.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.dm-0.in‚0 disk.drbd1938.in‚0 disk.rbd0.in‚0 disk.drbd1550.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1965.in‚0' proc.memory.availableMB‚ 38527.40625 disk.rbd1.in‚0 proc.memory.rssMB‚ 11253.3125 network.recv‚156717 disk.sda2.out‚20.71875 disk.drbd1584.in‚0 disk.drbd1965.out‚0 disk.sda.out‚20.71875 disk.drbd1226.in‚0 disk.drbd2111.out‚0 disk.drbd2111.in‚0 disk.drbd1552.out‚0 disk.rbd1.out‚ 0.46484375≈Gœ—‚(:‘ ¹œÄÓ­Îÿ¸ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚94.476' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚31.491999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜‚ÑÈB˜‚(:·  ¹œÄÓءǹ% memory_percent‚41.211843490600586 cpu‚14.155554064115162' proc.memory.availableMB‚ 38527.40625 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.sda.in‚0 disk.dm-0.in‚0* disk./.usagePercent‚31.895705162803058 disk.drbd1938.out‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1285.in‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 21.09765625 disk.drbd1356.out‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.46484375 disk.sda1.out‚0 disk.drbd1965.out‚0 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 proc.memory.rssMB‚ 11253.3125 network.recv‚159207 disk.drbd2111.in‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 21.09765625 proc.cpu.threads‚29$ disk./.usageGB‚570.3276023864746 disk.drbd1931.in‚0 disk.rbd0.in‚0 network.sent‚ 1.680194e+06 disk.dm-0.out‚24.47216796875 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd2041.out‚0 disk.drbd2118.in‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚0 disk.drbd1356.in‚0* proc.memory.percent‚17.171192169189453 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.drbd1510.in‚0# disk.nvme0n1.out‚24.47216796875 disk.sda2.in‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd1226.in‚0 disk.rbd0.out‚0—ð 0™+ _wandbruntime‚782  _runtime‚782t_ážÆšÀ & lr-AdamW/pg1‚4.507555416280036e-05 & lr-AdamW/pg2‚4.507555416280036e-05 trainer/global_step‚1749 % _timestamp‚1.7857941081353428e+09 _runtime‚ 782.212303552 _step‚136ïœp,˛Š& lr-AdamW/pg1‚4.507555416280036e-05 & lr-AdamW/pg2‚4.507555416280036e-05 trainer/global_step‚1749 % _timestamp‚1.7857941081353428e+09 _runtime‚ 782.212303552 _step‚136ˆ'ÐL0œ+ _wandbruntime‚782  _runtime‚782¸h¿¹'³O NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1539548521472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1535778902324 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11629829 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30106546 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18476719 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11909008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1581457017856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18476719 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 trainer/global_step‚1749 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5016863 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31585821996544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1539548521472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30835539336672 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30116204 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _runtime‚ 782.213661724 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11629827 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚31585821996544 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚30046360904704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29299846237472 _step‚137 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18486375 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30118303638528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11909257 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30118278734848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6892147 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29299863276832 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11629829 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30106546 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18476719 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11909008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1581457017856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18476719 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 trainer/global_step‚1749 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5016863 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31585821996544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1539548521472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30835539336672 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30116204 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _runtime‚ 782.213661724 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11629827 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚31585821996544 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚30046360904704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29299846237472 _step‚137 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18486375 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30118303638528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11909257 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30118278734848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6892147 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29299863276832 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30133384 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30853259824810 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18487242 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1536656375658 trainer/global_step‚1749 _step‚138 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11636472 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7857941085114644e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚31603972855296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1540340711424 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11915733 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30135510755328 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1540340711424 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30133384 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1582220363776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 782.58879973 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11636470 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11636472 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11636470 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1540428148224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚30063527667712 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30853156982214 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚31603868379136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30123712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11915973 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29316603449152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18487242 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚31603972855296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31717872271872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚30063527667712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1540428148224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18496912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚30063544707072 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6896084 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30063544707072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11636472 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7857941085114644e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚31603972855296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1540340711424 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11915733 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30135510755328 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1540340711424 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30133384 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1582220363776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 782.58879973 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11636470 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11636472 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11636470 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1540428148224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚30063527667712 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30853156982214 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚31603868379136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30123712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11915973 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29316603449152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18487242 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚31603972855296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31717872271872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚30063527667712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1540428148224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18496912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚30063544707072 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6896084 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30063544707072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30133384 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30853259824810 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18487242 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1536656375658 trainer/global_step‚1749 _step‚138 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E°r«¤ý 0loss/SRqqt1/detection_loss‚1.742363252403149 _runtime‚ 782.590088819 0loss/FRt1/assignment_loss‚3.7225301375082873 2loss/SRqqt1/assignment_loss‚1.4778324345238187 1loss/SRqqt2/detection_loss‚3.2261054141274577 epoch‚0 trainer/global_step‚1749 _step‚139 2loss/SRqqt2/assignment_loss‚2.2117841178324165 /loss/FRt1/detection_loss‚3.0870172182491067 .loss/FRt2/detection_loss‚4.459791242638055 /loss/FBt1/detection_loss‚22.695738080475067 $ _timestamp‚1.785794108513259e+09 .loss/FBt1/assignment_loss‚0.24335881213968 1loss/FBt2/assignment_loss‚0.14262820704607293 .loss/FBt2/detection_loss‚7.468594705514182 %loss/total_loss‚55.34164694509788 /loss/FRt2/assignment_loss‚4.863903322640588‹‘îN¬¥‚(:’ ÈœÄÓ«â㸠gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚123.143' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚41.047666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$™T@¦‚(:µ  ÈœÄө̬¹ disk.drbd1510.in‚0# disk.nvme0n1.out‚24.70263671875 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.sda.out‚ 21.6328125 cpu‚14.144295066848516 disk.drbd1550.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1356.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚0 disk.drbd1673.in‚0' proc.memory.availableMB‚ 38528.46875 proc.memory.rssMB‚ 11253.3125 disk.drbd1356.in‚0 disk.drbd1817.in‚0% memory_percent‚41.210222244262695 disk.rbd1.in‚0 disk.sda2.out‚ 21.6328125 disk.drbd1931.in‚0 disk.dm-0.out‚24.70263671875 disk.drbd1938.in‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1828.in‚0 disk.drbd2118.in‚0 disk.drbd1552.in‚0 disk.drbd1226.out‚0 disk.drbd1285.in‚0 disk.drbd1965.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.171192169189453* disk./.usagePercent‚31.895704096113853$ disk./.usageGB‚570.3275833129883 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1931.out‚0 disk.nvme0n1.in‚0 disk.drbd1552.out‚0 network.sent‚ 1.683261e+06 network.recv‚160304 disk.drbd1285.out‚0 disk.rbd1.out‚ 0.46484375 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.sda1.in‚0 disk.drbd1226.in‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1673.out‚0 proc.cpu.threads‚29 disk.drbd1510.out‚0 disk.sda2.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0’[wD0§+ _wandbruntime‚804  _runtime‚804$ìJcƨÀ trainer/global_step‚1799 % _timestamp‚1.7857941302745366e+09 _runtime‚ 804.351523473 _step‚140 & lr-AdamW/pg1‚4.636416348706567e-05 & lr-AdamW/pg2‚4.636416348706567e-05ÇäӬ˩Š& lr-AdamW/pg1‚4.636416348706567e-05 & lr-AdamW/pg2‚4.636416348706567e-05 trainer/global_step‚1799 % _timestamp‚1.7857941302745366e+09 _runtime‚ 804.351523473 _step‚140Œˆ,±®0ª+ _wandbruntime‚804  _runtime‚804ϘÚá¹'«³O SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5159177 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚32488156282368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚1799 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚32488260751360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1579667365832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1579581562708 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19004721 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 804.352750108 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1626641183744 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19014377 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11961979 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32605149278720 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7857941302755382e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11961979 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30136854853472 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚32488156282368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5159425 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1583457227264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19004721 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚32488260751360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1583457227264 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30904716094464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30976356 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12248172 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1626514703872|µÌ­¬¸O N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12248172 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1626514703872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5159177 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚32488156282368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚1799 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚32488260751360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1579667365832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1579581562708 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19004721 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 804.352750108 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1626641183744 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19014377 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11961979 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32605149278720 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7857941302755382e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11961979 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30136854853472 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚32488156282368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5159425 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1583457227264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19004721 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚32488260751360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1583457227264 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30904716094464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30976356 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640ò¯k0­+ _wandbruntime‚804  _runtime‚804 ¹ÞÉÂ&®¼M @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11968622 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30995841691648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30153595025792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19015244 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30983864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7092932 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5162298 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚30921865818112 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19024914 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11968620 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19024914 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11968620 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32623412072960 trainer/global_step‚1799 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _runtime‚ 804.720938053 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚30921882857472 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7092934 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚31734156904318 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1580459035930 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚142 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚32506307141120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19015244 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1580544839166 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚30921865818112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30921882857472 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30995866595328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚31734054061722 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12254977 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚32506202664960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12255232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30993536 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5162045 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30983864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392ÔNçµÇ&¯ÁM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7092932 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5162298 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚30921865818112 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19024914 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11968620 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19024914 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11968620 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32623412072960 trainer/global_step‚1799 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _runtime‚ 804.720938053 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚30921882857472 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7092934 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚31734156904318 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1580459035930 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚142 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚32506307141120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19015244 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1580544839166 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚30921865818112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30921882857472 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30995866595328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚31734054061722 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12254977 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚32506202664960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12255232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30993536 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5162045 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30983864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11968622 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30995841691648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30153595025792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19015244 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30983864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936ŽýÄL 0°+ _wandbruntime‚804  _runtime‚804]òŸý±÷ /loss/FRt2/assignment_loss‚4.961658382415772 2loss/SRqqt1/assignment_loss‚1.6279451372783358 .loss/SRqqt1/detection_loss‚2.0395393420845 1loss/SRqqt2/detection_loss‚2.6460163683447417 %loss/total_loss‚45.31629000449302 _runtime‚ 804.722405928 _step‚143 1loss/FBt1/assignment_loss‚0.35304782736264834 .loss/FRt1/detection_loss‚3.422271540543685 .loss/FBt2/detection_loss‚6.275074821853747 trainer/global_step‚1799 1loss/FBt2/assignment_loss‚0.07429395777616286 .loss/FRt2/detection_loss‚4.668956447564638 /loss/FBt1/detection_loss‚13.228407898119517 $ _timestamp‚1.785794130645629e+09 0loss/FRt1/assignment_loss‚3.9866522774100304 1loss/SRqqt2/assignment_loss‚2.032426003739238 epoch‚0l‰f˲ü 2loss/SRqqt1/assignment_loss‚1.6279451372783358 .loss/SRqqt1/detection_loss‚2.0395393420845 1loss/SRqqt2/detection_loss‚2.6460163683447417 %loss/total_loss‚45.31629000449302 _runtime‚ 804.722405928 _step‚143 1loss/FBt1/assignment_loss‚0.35304782736264834 .loss/FRt1/detection_loss‚3.422271540543685 .loss/FBt2/detection_loss‚6.275074821853747 trainer/global_step‚1799 1loss/FBt2/assignment_loss‚0.07429395777616286 .loss/FRt2/detection_loss‚4.668956447564638 /loss/FBt1/detection_loss‚13.228407898119517 $ _timestamp‚1.785794130645629e+09 0loss/FRt1/assignment_loss‚3.9866522774100304 1loss/SRqqt2/assignment_loss‚2.032426003739238 epoch‚0 /loss/FRt2/assignment_loss‚4.961658382415772lîz©³‚(:’ לÄÓýºå¸ gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚125.473' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚41.824333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓnA´‚(:¶  לÄÓ‚ÿ«¹ disk.drbd2078.in‚0 disk.drbd2111.in‚0* disk./.usagePercent‚31.895704096113853 disk.drbd1285.out‚0 disk.sda.out‚ 21.88671875 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.nvme0n1.in‚0$ disk./.usageGB‚570.3275833129883 disk.drbd2041.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1226.out‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 proc.cpu.threads‚29 disk.drbd1965.out‚0* proc.memory.availableMB‚38528.18359375 network.sent‚ 1.728195e+06 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.sda2.in‚0% memory_percent‚41.210657358169556 disk.drbd1965.in‚0 disk.drbd1289.out‚0 disk.sda1.in‚0! disk.nvme0n1.out‚ 25.080078125 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 0.46484375 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd1285.in‚0 disk.drbd1931.out‚0 disk.dm-0.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0 network.recv‚162915 disk.drbd1938.out‚0 disk.drbd1817.in‚0 disk.drbd1552.out‚0 disk.drbd1289.in‚0* proc.memory.percent‚17.171192169189453 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.dm-0.out‚ 25.080078125 proc.memory.rssMB‚ 11253.3125 disk.sda.in‚0 disk.drbd1931.in‚0 disk.drbd2111.out‚0 cpu‚14.139260176899187 disk.drbd1938.in‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 21.88671875C=£ó‘µ‚(:† æœÄÓâ™å¸ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚124.776' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚41.592 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì]{]?¶‚(:´  æœÄÓÐ) proc.memory.availableMB‚ 38528.4140625 disk.drbd2078.out‚0 proc.cpu.threads‚29 disk.drbd2111.in‚0 disk.drbd1965.out‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.nvme0n1.in‚0 disk.rbd0.out‚0 disk.drbd1552.out‚0 disk.sda.out‚ 22.32421875 disk.drbd1938.in‚0* proc.memory.percent‚17.171192169189453 disk.rbd0.in‚0 disk.drbd2078.in‚0 disk.drbd1673.in‚0 disk.drbd1550.out‚0! disk.nvme0n1.out‚ 25.232421875 disk.drbd1356.out‚0 network.sent‚ 1.773187e+06 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.drbd1931.in‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.sda2.out‚ 22.32421875 disk.drbd1289.out‚0 disk.rbd1.out‚ 0.46484375 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.sda2.in‚0 disk.drbd1289.in‚0 disk.dm-0.in‚0 disk.dm-0.out‚ 25.232421875 disk.drbd1817.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda1.in‚0* disk./.usagePercent‚31.895704309451695$ disk./.usageGB‚570.3275871276855 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 proc.memory.rssMB‚ 11253.3125 disk.drbd1828.in‚0 disk.drbd2118.out‚0 disk.drbd1510.in‚0$ memory_percent‚41.21030569076538 cpu‚14.133694210270871 disk.drbd1965.in‚0 network.recv‚165456 disk.drbd2041.out‚0 disk.drbd1510.out‚0T0·+ _wandbruntime‚826  _runtime‚826_zYƸÀ & lr-AdamW/pg1‚4.765277281133098e-05 & lr-AdamW/pg2‚4.765277281133098e-05 trainer/global_step‚1849 % _timestamp‚1.7857941524919226e+09 _runtime‚ 826.568865282 _step‚144L‚Û˹Š& lr-AdamW/pg1‚4.765277281133098e-05 & lr-AdamW/pg2‚4.765277281133098e-05 trainer/global_step‚1849 % _timestamp‚1.7857941524919226e+09 _runtime‚ 826.568865282 _step‚144NöÖí0º+ _wandbruntime‚826  _runtime‚826 (HÇh»³O ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_po%afQ&ol.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31836508 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚32597436338172 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 trainer/global_step‚1849 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31826850 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚33510851985408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1627540792320 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5302478 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1671699701248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1623555829340 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19532723 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31838990414848 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _runtime‚ 826.570143186 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31763054244864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚33390595037184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12588586 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚31763037205504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19542379 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1671836666880 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31839015318528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12294127 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7285845 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 % _timestamp‚1.7857941524929168e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚33510690116096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1627453362688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31763037205504 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚31763054244864 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33390490568192 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚32597333495688 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19532723 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19542379ýu˜¾'¼¸O SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5302478 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1671699701248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1623555829340 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19532723 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31838990414848 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _runtime‚ 826.570143186 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31763054244864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚33390595037184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12588586 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚31763037205504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19542379 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1671836666880 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31839015318528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12294127 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7285845 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 % _timestamp‚1.7857941524929168e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚33510690116096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1627453362688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31763037205504 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚31763054244864 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33390490568192 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚32597333495688 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19532723 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19542379 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31836508 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚32597436338172 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 trainer/global_step‚1849 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31826850 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚33510851985408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1627540792320‘4ÀÈ(0½+ _wandbruntime‚826  _runtime‚8264²Ã&¾½M QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1628332982272 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5305374 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1624347499438 % _timestamp‚1.7857941528571713e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31844016 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31844016 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚32615053983826 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1628332982272 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1672606943232 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33408641426944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31780221007872 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚33528957948416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31856222435328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31856197531648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12595156 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19552916 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7289784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31853688 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12300772 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 _step‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31853688 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12300770 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚33528804474880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚33408641426944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1628420419072 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12300770 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19552916 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31780221007872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30990620681152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1849 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1628420419072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7289782 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚32614951141230 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19543246 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12300772 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚31780203968512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12300772 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚31780203968512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31844016 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚32615053983826 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1628332982272 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1672606943232 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33408641426944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31780221007872 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚33528957948416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31856222435328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31856197531648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12595156 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19552916 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7289784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31853688 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12300772 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 _step‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31853688 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12300770 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚33528804474880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚33408641426944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1628420419072 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12300770 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19552916 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31780221007872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30990620681152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1849 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1628420419072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7289782 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚32614951141230 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0’Eèö%0À+ _wandbruntime‚826  _runtime‚826y ÐÞýÁ÷ _step‚147 1loss/SRqqt1/detection_loss‚1.7173585415448689 $loss/total_loss‚48.8655363320439 # _timestamp‚1.78579415285898e+09 /loss/FRt2/assignment_loss‚4.838444769382477 2loss/SRqqt1/assignment_loss‚1.6927126405239106 0loss/FBt1/assignment_loss‚0.5740476887334477 /loss/FRt1/detection_loss‚2.8025619632158523 .loss/FBt1/detection_loss‚20.80591444535689 epoch‚0 trainer/global_step‚1849 0loss/FRt1/assignment_loss‚3.5759673067647166 2loss/SRqqt2/assignment_loss‚2.2661580192415336 1loss/FBt2/assignment_loss‚0.11617315951777372 .loss/FRt2/detection_loss‚4.432138088871451 0loss/SRqqt2/detection_loss‚2.669472854903114 .loss/FBt2/detection_loss‚3.374586853987865 _runtime‚ 826.935769804ù_ý¹Âü 0loss/FRt1/assignment_loss‚3.5759673067647166 2loss/SRqqt2/assignment_loss‚2.2661580192415336 1loss/FBt2/assignment_loss‚0.11617315951777372 .loss/FRt2/detection_loss‚4.432138088871451 0loss/SRqqt2/detection_loss‚2.669472854903114 .loss/FBt2/detection_loss‚3.374586853987865 _runtime‚ 826.935769804 _step‚147 1loss/SRqqt1/detection_loss‚1.7173585415448689 $loss/total_loss‚48.8655363320439 # _timestamp‚1.78579415285898e+09 /loss/FRt2/assignment_loss‚4.838444769382477 2loss/SRqqt1/assignment_loss‚1.6927126405239106 0loss/FBt1/assignment_loss‚0.5740476887334477 /loss/FRt1/detection_loss‚2.8025619632158523 .loss/FBt1/detection_loss‚20.80591444535689 epoch‚0 trainer/global_step‚1849“Ê5[œÃ‚(:‘ õœÄÓ€ÝḠgpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚127.814' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚42.60466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Â'BÄ‚(:·  õœÄÓ¾ù©¹* proc.memory.percent‚17.171192169189453 disk.drbd1938.out‚0 disk.drbd1289.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 22.5859375 disk.drbd2041.out‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd1938.in‚0 disk.drbd1817.out‚0 disk.drbd1356.out‚0 disk.drbd1817.in‚0# disk.nvme0n1.out‚25.54931640625 disk.drbd1285.in‚0 disk.sda2.in‚0 proc.memory.rssMB‚ 11253.3125% memory_percent‚41.211384534835815) disk./.usagePercent‚31.89567700220806 disk.drbd1828.in‚0 disk.sda.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.drbd2118.in‚0 disk.rbd1.out‚ 0.46484375 disk.drbd1584.out‚0 disk.drbd1226.out‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd2041.in‚0 cpu‚14.123900076643698 network.sent‚ 1.776212e+06 disk.drbd1584.in‚0 disk.sda1.out‚0 proc.cpu.threads‚29 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.drbd1552.out‚0 disk.rbd0.in‚0 disk.drbd1550.in‚0* proc.memory.availableMB‚38527.70703125$ disk./.usageGB‚570.3270988464355 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.dm-0.out‚25.54931640625 disk.drbd1965.in‚0 disk.sda2.out‚ 22.5859375 disk.drbd1931.out‚0 disk.dm-0.in‚0 disk.nvme0n1.in‚0 disk.drbd1285.out‚0 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1510.out‚0 network.recv‚166405 disk.drbd1356.in‚07ɦ“0Å+ _wandbruntime‚848  _runtime‚848ˆ™çáÇÆÁ ' lr-AdamW/pg2‚4.8941382135596276e-05 trainer/global_step‚1899 $ _timestamp‚1.785794174776713e+09 _runtime‚ 848.853655326 _step‚148 ' lr-AdamW/pg1‚4.8941382135596276e-05èÑ1ÌÇÆ ' lr-AdamW/pg1‚4.8941382135596276e-05 ' lr-AdamW/pg2‚4.8941382135596276e-05 trainer/global_step‚1899 $ _timestamp‚1.785794174776713e+09 _runtime‚ 848.853655326 _step‚148”-¾`y0È+ _wandbruntime‚848  _runtime‚848;~ȹ'ɳO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32687002 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚34292824854016 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5445673 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34292824854016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 % _timestamp‚1.7857941747777781e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32621375355904 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 _step‚149 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12626279 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12928135 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1667358489724 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12626277 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12626277 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12928370 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12626279 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1671449498112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20060725 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7482697 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32696660 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚34292929323008 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚32621375355904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5445440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34416211240960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1667444292848 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 trainer/global_step‚1899 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31810872085472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20060725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1671536927744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚34292929323008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1716999854592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚32621392395264 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20070381 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚32699371158528 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032m‘"U¾'ʸO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7482695 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1671536927744 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚34292824854016 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5445673 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34292824854016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 % _timestamp‚1.7857941747777781e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32621375355904 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 _step‚149 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12626279 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12928135 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1667358489724 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12626277 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12626277 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12928370 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12626279 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1671449498112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20060725 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7482697 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32696660 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚34292929323008 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚32621375355904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5445440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34416211240960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1667444292848 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 trainer/global_step‚1899 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31810872085472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20060725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1671536927744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚34292929323008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1716999854592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚32621392395264 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20070381 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚32699371158528 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32687002 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚34310871236608 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7486632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1717897857024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 trainer/global_step‚1899 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32713840 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32713840 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7486634 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚31827612257792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚32638559158272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5448488 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚34310975712768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚32638542118912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1717767190016 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12632922 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12934870 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20080918 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 849.221793174 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5448238 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12632920 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32704168 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚34310975712768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12632922 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20071248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1668235962946 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚32716553371648 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚33495951063334 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚32638542118912 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1672329117696 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20071248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12935122 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1672329117696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20080918 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34434476132352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚33495848220738 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130_vÂÈ&ÍÂM trainer/global_step‚1899 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32713840 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32713840 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7486634 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚31827612257792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚32638559158272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5448488 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚34310975712768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚32638542118912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1717767190016 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12632922 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12934870 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20080918 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 849.221793174 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5448238 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12632920 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32704168 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚34310975712768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12632922 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20071248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1668235962946 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚32716553371648 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚33495951063334 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚32638542118912 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1672329117696 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20071248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12935122 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1672329117696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20080918 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34434476132352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚33495848220738 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 % _timestamp‚1.7857941751444452e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚34310871236608 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7486632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1717897857024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032–ü3âu0Î+ _wandbruntime‚849  _runtime‚849ؤþÏø 0loss/FBt2/assignment_loss‚0.0866946679104303 1loss/SRqqt1/detection_loss‚1.9843866031152284 /loss/FBt2/detection_loss‚6.0804545081076995 0loss/FRt1/assignment_loss‚3.5963489447182755 .loss/FRt2/detection_loss‚5.408144491138281 -loss/FBt1/detection_loss‚123.477270424366 trainer/global_step‚1899 % _timestamp‚1.7857941751462228e+09 /loss/FRt2/assignment_loss‚5.219699400442618 0loss/SRqqt1/assignment_loss‚1.70045119535923 2loss/SRqqt2/assignment_loss‚1.9615851357215788 1loss/FBt1/assignment_loss‚0.48702630400657654 .loss/FRt1/detection_loss‚2.701691917953937 1loss/SRqqt2/detection_loss‚2.3074550309320148 epoch‚0 _step‚151 %loss/total_loss‚155.0112086237719 _runtime‚ 849.222997666LúfÐý epoch‚0 _step‚151 %loss/total_loss‚155.0112086237719 _runtime‚ 849.222997666 0loss/FBt2/assignment_loss‚0.0866946679104303 1loss/SRqqt1/detection_loss‚1.9843866031152284 /loss/FBt2/detection_loss‚6.0804545081076995 0loss/FRt1/assignment_loss‚3.5963489447182755 .loss/FRt2/detection_loss‚5.408144491138281 -loss/FBt1/detection_loss‚123.477270424366 trainer/global_step‚1899 % _timestamp‚1.7857941751462228e+09 /loss/FRt2/assignment_loss‚5.219699400442618 0loss/SRqqt1/assignment_loss‚1.70045119535923 2loss/SRqqt2/assignment_loss‚1.9615851357215788 1loss/FBt1/assignment_loss‚0.48702630400657654 .loss/FRt1/detection_loss‚2.701691917953937 1loss/SRqqt2/detection_loss‚2.3074550309320148—mܶºœÑ‚(:‘ „ÄÓ—¥˜¹ gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚124.502' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.50066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Fã&CÒ‚(:¸  „ÄÓæ¥Þ¹ disk.dm-0.in‚0 disk.drbd1938.out‚0 proc.cpu.threads‚29 disk.drbd1817.in‚0 disk.drbd1356.in‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0% memory_percent‚41.211819648742676 disk.drbd1817.out‚0 disk.drbd1285.out‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.nvme0n1.in‚0 disk.drbd1285.in‚0 disk.drbd1673.in‚0* proc.memory.percent‚17.171192169189453 network.recv‚168886 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0* disk./.usagePercent‚31.895787511209623 disk.drbd1938.in‚0# disk.nvme0n1.out‚25.74462890625 disk.drbd1552.out‚0 disk.sda2.out‚ 22.91796875 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 22.91796875$ disk./.usageGB‚570.3290748596191 disk.drbd1965.out‚0 disk.drbd1289.out‚0 disk.sda1.in‚0 disk.drbd1828.in‚0 disk.dm-0.out‚25.74462890625 proc.memory.rssMB‚ 11253.3125 disk.drbd1226.in‚0 disk.rbd1.out‚ 0.46484375 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.drbd1226.out‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.drbd2041.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.sda2.in‚0 disk.drbd1550.out‚0 disk.drbd1931.in‚0 disk.drbd1584.in‚0 network.sent‚ 1.821098e+06 disk.drbd1510.in‚0( proc.memory.availableMB‚ 38527.421875 cpu‚14.114954208345162 disk.drbd2078.in‚0]ÏÞ5‘Ó‚(:† “ÄÓšŽø¸ gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚123.825' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚41.275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è˜EºGÔ‚(:¼  “ÄÓÿŠÀ¹ disk.rbd1.out‚ 0.46484375* proc.memory.percent‚17.171233892440796 disk.drbd2111.out‚0* proc.memory.availableMB‚38527.62890625 network.sent‚ 1.865994e+06$ disk./.usageGB‚570.3286285400391 network.recv‚171139 disk.drbd2118.in‚0 disk.drbd1285.in‚0% memory_percent‚41.211503744125366 disk.drbd1938.in‚0 disk.drbd1510.in‚0 disk.rbd0.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.sda.out‚ 23.3671875 disk.dm-0.out‚25.92041015625 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1938.out‚0 disk.nvme0n1.in‚0 disk.rbd1.in‚0 disk.sda2.out‚ 23.3671875 disk.sda.in‚0 disk.drbd1817.in‚0 disk.sda2.in‚0 disk.dm-0.in‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd1285.out‚0* disk./.usagePercent‚31.895762550682242 disk.sda1.out‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd1356.in‚0 disk.drbd1965.out‚0 disk.rbd0.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1550.out‚0 proc.cpu.threads‚29 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0$ proc.memory.rssMB‚11253.33984375 disk.drbd1931.out‚0 disk.drbd1673.in‚0# disk.nvme0n1.out‚25.92041015625 disk.drbd1356.out‚0 disk.drbd1289.in‚0 cpu‚14.106448729956622 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1817.out‚0 disk.drbd1552.in‚0œë}10Õ+ _wandbruntime‚870  _runtime‚870îF¨ÅÖ¿ _runtime‚ 870.459864823 _step‚152 & lr-AdamW/pg1‚5.022999145986158e-05 & lr-AdamW/pg2‚5.022999145986158e-05 trainer/global_step‚1949 $ _timestamp‚1.785794196382898e+09“‡ªÄÊ×Ä trainer/global_step‚1949 $ _timestamp‚1.785794196382898e+09 _runtime‚ 870.459864823 _step‚152 & lr-AdamW/pg1‚5.022999145986158e-05 & lr-AdamW/pg2‚5.022999145986158e-05˜†œ»Û0Ø+ _wandbruntime‚870  _runtime‚870ù©ªFÙ²O G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12958427 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚32647880701472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20588727 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚35195263608832 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1715533063168 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35195159139840 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7679547 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1762170735104 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚153 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1715533063168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 _runtime‚ 870.46120331 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656—ï½'Ú·O TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12958429 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13268212 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5588922 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1711332756356 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33559726998528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33559702094848 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7679545 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12958427 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚32647880701472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20588727 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚35195263608832 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1715533063168 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35195159139840 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7679547 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1762170735104 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚153 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1715533063168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 _runtime‚ 870.46120331 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12958427 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33496897308672 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12965072 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚32664637913152 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20599250 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7857941967500408e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚1949 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20599250 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚35213309998592 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33496880269312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1762935023104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33573992 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚35213309998592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1716412689920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33573992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1716325253120 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20608920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5591761 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33576934115328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _runtime‚ 870.827380477 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33564320 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13275010 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚33496880269312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13275245 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12965072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12965070 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7683482 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1712210229690 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12965070 _step‚154 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚34376848142842 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1716412689920 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7683484 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1716325253120 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888Nœ7"È&ÝÂM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚34376745300246 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33576909211648 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1712124426454 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5591528 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚34376848142842 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1716412689920 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7683484 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1716325253120 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1763071981568 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33496897308672 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12965072 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚32664637913152 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20599250 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7857941967500408e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚1949 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20599250 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚35213309998592 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33496880269312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1762935023104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33573992 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚35213309998592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1716412689920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33573992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1716325253120 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20608920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5591761 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33576934115328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _runtime‚ 870.827380477 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33564320 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13275010 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚33496880269312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13275245 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12965072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12965070 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7683482 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1712210229690 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12965070 _step‚154 ODeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13608996 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1807163984384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚36097493425664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1759441768960 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5732599 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34407306 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21116729 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1999 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚34420082838528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚35240127576696 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1755135416740 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13290577 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚34338068696064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1759529198592 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130)ä.V¾'è¸O ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚36097493425664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1759441768960 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5732599 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34407306 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21116729 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1999 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚34420082838528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚35240127576696 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1755135416740 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13290577 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚34338068696064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1759529198592 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21116729 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚34338051656704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13290577 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚34338051656704 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚33484906356832 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚35240024734212 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7876397 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1759441768960 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13608737 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚36097597894656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1759529198592 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21126385 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5732342 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚36097597894656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21126385 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚ 892.893503419 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36227221919232 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34420057934848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1807294658560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7876395 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚33484889317472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1755221219864 % _timestamp‚1.7857942188162394e+09 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13608996 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1807163984384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18&݈l0é+ _wandbruntime‚893  _runtime‚893,hé ê½M REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21136922 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1756012889962 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21136922 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 _runtime‚ 893.265072108 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 % _timestamp‚1.7857942191877065e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚35257745222350 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1999 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚35257642379754 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5735149 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13297222 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1760321388544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚158 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1760321388544 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34424472 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚34355235459072 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13297222 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33501629489792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34434144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1760408825344 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34424472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33501646529152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21127252 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13615481 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1808078795776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚34355235459072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36245503612416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7880334 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚36115539808256 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21127252 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34434144%Ë­pÈ&ëÂM S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚36115644284416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13615726 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚34355235459072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36245503612416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7880334 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚36115539808256 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21127252 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34434144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21136922 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1756012889962 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21136922 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 _runtime‚ 893.265072108 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 % _timestamp‚1.7857942191877065e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚35257745222350 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1999 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚35257642379754 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5735149 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13297222 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1760321388544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚158 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1760321388544 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34424472 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚34355235459072 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13297222 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33501629489792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34434144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1760408825344 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34424472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33501646529152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21127252 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13615481 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1808078795776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚36999827711488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1803525334016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚36999932180480 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1803437904384 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5875065 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35277116 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1799109683372 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36120921813720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13622729 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚35196389807104 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5875323 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚36999827711488 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1799023880248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35280413774848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 trainer/global_step‚2049 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1803437904384 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚34321897933472 % _timestamp‚1.7857942409101121e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37132936674304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚35196406846464 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37132774804992 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1852497995776 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1852361030144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36121024656204 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚35196406846464 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚34321914972832 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚36999932180480 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13622729 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21654387 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21654387 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1803525334016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13948570 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8073247 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _runtime‚ 914.987339316ü÷U)Mö¸O WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1803437904384 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5875065 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35277116 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1799109683372 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36120921813720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13622729 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚35196389807104 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5875323 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚36999827711488 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1799023880248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35280413774848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 trainer/global_step‚2049 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1803437904384 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚34321897933472 % _timestamp‚1.7857942409101121e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37132936674304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚35196406846464 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37132774804992 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1852497995776 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1852361030144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36121024656204 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚35196406846464 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚34321914972832 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚36999932180480 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13622729 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21654387 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21654387 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1803525334016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13948570 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8073247 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _runtime‚ 914.987339316 E8D3:+q eviceStatsMonitor.on_train_batch_start/active.all.freed‚35267458 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚36999827711488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1803525334016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚36999932180480¡«™táœ÷‚(:‘ ÀÄÓ‰‚„¹ gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚127.781' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚42.59366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tøÞ^Bø‚(:·  ÀÄÓÁàÒ¹ network.recv‚176849 disk.rbd0.out‚0 disk.dm-0.in‚0" disk.nvme0n1.out‚ 26.7353515625$ disk./.usageGB‚570.3286094665527 disk.drbd2111.in‚0 disk.drbd1931.in‚0 disk.drbd1552.out‚0 disk.drbd1965.in‚0 network.sent‚ 1.959008e+06 disk.drbd1584.out‚0 disk.drbd1510.out‚0 cpu‚14.080938205979578 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.drbd1817.out‚0 disk.drbd2078.in‚0 disk.drbd2118.in‚0 disk.drbd1828.out‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.nvme0n1.in‚0 disk.drbd1285.out‚0* proc.memory.availableMB‚38526.78515625 disk.drbd1584.in‚0 disk.sda.out‚ 24.390625 disk.drbd1817.in‚0 disk.drbd1356.in‚0 proc.cpu.threads‚29 disk.drbd1285.in‚0 disk.rbd1.out‚ 0.46484375 disk.sda2.out‚ 24.390625 disk.sda1.out‚0 disk.drbd1938.in‚0 disk.drbd1552.in‚0 disk.rbd0.in‚0 disk.drbd2078.out‚0 disk.sda.in‚0 disk.drbd1510.in‚0 disk.drbd1356.out‚0 disk.drbd1289.out‚0 disk.drbd2118.out‚0 disk.sda1.in‚0 disk.dm-0.out‚ 26.7353515625 disk.drbd1938.out‚0 disk.drbd1673.out‚0 disk.drbd1226.out‚0 disk.sda2.in‚0 disk.drbd1550.in‚0$ proc.memory.rssMB‚11253.33984375 disk.drbd1673.in‚0 disk.drbd1550.out‚0 disk.drbd2111.out‚0 disk.drbd2041.in‚0% memory_percent‚41.212791204452515) disk./.usagePercent‚31.89576148399304 disk.drbd1931.out‚0* proc.memory.percent‚17.171233892440796oò0ù+ _wandbruntime‚915  _runtime‚915ç•3sÃ&ú½M ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚34338655145152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13955079 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1799987156706 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚34338638105792 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21655254 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚35213573609472 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35284624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35297620891648 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1804317523968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36138539459262 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21655254 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚35213573609472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13629370 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35294296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13629370 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚2049 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1799901353470 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1804404960768 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8077184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8077182 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 _step‚162 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 % _timestamp‚1.7857942412770352e+09 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35294296 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35284624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚37150889622528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21664924 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21664924 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1804317523968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1804404960768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35297645795328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37017874094080 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13629372 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚35213556570112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 915.354326286 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13629372 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5878142 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37151043096064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1853397300736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1853268730880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5877897 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚36138642301858 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13955079 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1799987156706 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚34338638105792 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21655254 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚35213573609472 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35284624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35297620891648 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1804317523968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36138539459262 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21655254 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚35213573609472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13629370 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35294296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13629370 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚2049 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1799901353470 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1804404960768 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8077184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8077182 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 _step‚162 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 % _timestamp‚1.7857942412770352e+09 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35294296 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35284624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚37150889622528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21664924 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21664924 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1804317523968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1804404960768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35297645795328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37017874094080 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13629372 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚35213556570112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 915.354326286¢¶Oäå0ü+ _wandbruntime‚915  _runtime‚915 ýú % _timestamp‚1.7857942412791147e+09 %loss/total_loss‚53.65889980114968 1loss/FBt2/assignment_loss‚0.08010361317719798 2loss/SRqqt2/assignment_loss‚1.8169746234499176 0loss/FBt1/assignment_loss‚0.5961260742374829 _runtime‚ 915.355929843 _step‚163 epoch‚0 /loss/FRt1/assignment_loss‚3.749618727881629 1loss/SRqqt1/assignment_loss‚1.736098522113429 /loss/FRt1/detection_loss‚2.8774197143208875 1loss/SRqqt1/detection_loss‚2.0811891907737508 1loss/SRqqt2/detection_loss‚2.8649257157927925 /loss/FBt1/detection_loss‚23.837238077606475 .loss/FBt2/detection_loss‚4.942315949475948 /loss/FRt2/assignment_loss‚4.759958526981411 .loss/FRt2/detection_loss‚4.316931065338761 trainer/global_step‚2049…ÞÐöþÿ /loss/FBt1/detection_loss‚23.837238077606475 .loss/FBt2/detection_loss‚4.942315949475948 /loss/FRt2/assignment_loss‚4.759958526981411 .loss/FRt2/detection_loss‚4.316931065338761 trainer/global_step‚2049 % _timestamp‚1.7857942412791147e+09 %loss/total_loss‚53.65889980114968 1loss/FBt2/assignment_loss‚0.08010361317719798 2loss/SRqqt2/assignment_loss‚1.8169746234499176 0loss/FBt1/assignment_loss‚0.5961260742374829 _runtime‚ 915.355929843 _step‚163 epoch‚0 /loss/FRt1/assignment_loss‚3.749618727881629 1loss/SRqqt1/assignment_loss‚1.736098522113429 /loss/FRt1/detection_loss‚2.8774197143208875 1loss/SRqqt1/detection_loss‚2.0811891907737508 1loss/SRqqt2/detection_loss‚2.8649257157927925£e˜Tÿ‚(:’ ÏÄӳ߸ gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚122.648' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.882666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßòzK@€‚(:µ  ÏÄÓœ—§¹ disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1673.in‚0 disk.drbd1285.out‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd2041.out‚0" disk.nvme0n1.out‚ 26.9580078125 disk.drbd1550.out‚0 proc.cpu.threads‚29 disk.dm-0.in‚0 disk.drbd1965.out‚0 disk.sda.in‚0 disk.sda.out‚ 24.6328125 disk.drbd2041.in‚0$ proc.memory.rssMB‚11253.33984375 network.recv‚179081 disk.nvme0n1.in‚0 disk.drbd1285.in‚0 disk.rbd1.in‚0 disk.drbd2111.out‚0 disk.drbd1931.in‚0 disk.drbd1552.in‚0 disk.drbd1965.in‚0 disk.rbd1.out‚ 0.46484375 disk.rbd0.in‚0 disk.drbd1289.in‚0 disk.drbd2111.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0 disk.sda1.out‚0 disk.drbd2078.out‚0* proc.memory.percent‚17.171233892440796# memory_percent‚41.2121057510376 disk.sda2.out‚ 24.6328125 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd1584.in‚0) disk./.usagePercent‚31.89566164188352 disk.rbd0.out‚0 disk.drbd1828.out‚0 disk.drbd1584.out‚0 cpu‚14.073208729796342 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1817.out‚0 disk.drbd1550.in‚0 disk.drbd1931.out‚0 disk.drbd1510.in‚0 disk.sda2.in‚0 network.sent‚ 1.993269e+06$ disk./.usageGB‚570.3268241882324 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 38527.234375 disk.dm-0.out‚ 26.9580078125I?´U0+ _wandbruntime‚936  _runtime‚936ê¸w‚¼ % lr-AdamW/pg1‚5.40958194326575e-05 % lr-AdamW/pg2‚5.40958194326575e-05 trainer/global_step‚2099 $ _timestamp‚1.785794262793257e+09 _runtime‚ 936.87023904 _step‚164觤¹ÇƒÁ _runtime‚ 936.87023904 _step‚164 % lr-AdamW/pg1‚5.40958194326575e-05 % lr-AdamW/pg2‚5.40958194326575e-05 trainer/global_step‚2099 $ _timestamp‚1.785794262793257e+09¤ùXB0„+ _wandbruntime‚936  _runtime‚936øBýQ‹ …³O O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36137268 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13954879 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14288623 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1847521469440 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38038301885952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6018528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37902161997312 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1847521469440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22172733 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1842912343756 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37902266466304 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1847434039808 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13954877 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36054727957504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3515BH}4.8906549472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22182389 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14288861 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13954877 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22182389 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35158923588832 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8270095 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1897667139584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37902266466304 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8270097 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36127610 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _runtime‚ 936.871609526 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36054727957504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36054744996864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚37001921735712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚37001818893228 % _timestamp‚1.7857942627943473e+09 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37902161997312 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚38038461658112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13954879 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1842998146880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36054744996864 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1847521469440 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38038301885952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6018528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37902161997312 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1847521469440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22172733 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1842912343756 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37902266466304 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1847434039808 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13954877 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36054727957504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35158906549472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22182389 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14288861 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13954877 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22182389 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35158923588832 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8270095 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1897667139584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37902266466304 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8270097 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36127610 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _runtime‚ 936.871609526 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36054727957504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936¥˜²6Y0‡+ _wandbruntime‚937  _runtime‚937;tgûÃ&ˆ½M S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37920312856064 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚38056583620096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37920312856064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37920208379904 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚166 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6021335 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 % _timestamp‚1.7857942631683655e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1848313659392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36154448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1898445026304 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36071911759872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36071894720512 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1898581984768 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36157976731648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13961520 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14295367 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1843789816978 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6021591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13961520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚2099 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8274032 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36154448 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36071911759872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38056421757952xÆþ‰ÂM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13961520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚2099 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8274032 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36154448 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36071911759872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38056421757952 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37920312856064 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚38056583620096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37920312856064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37920208379904 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚166 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6021335 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 % _timestamp‚1.7857942631683655e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1848313659392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36154448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1898445026304 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36071911759872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36071894720512 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1898581984768 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36157976731648 ;4DeviceStatsMonitor.on_train_batch_end/max_s`§mÊplit_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13961520 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14295367 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1843789816978 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6021591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¦‚Åð³0Š+ _wandbruntime‚937  _runtime‚937ÕbþŒ‹ý 1loss/SRqqt1/detection_loss‚1.7016665346373985 %loss/total_loss‚46.91370846942982 trainer/global_step‚2099 _runtime‚ 937.246829711 _step‚167 0loss/FRt1/assignment_loss‚3.4704618337107638 2loss/SRqqt2/assignment_loss‚1.7836387367584767 2loss/FBt2/assignment_loss‚0.061299103559576906 .loss/FBt2/detection_loss‚4.985289855280671 2loss/SRqqt1/assignment_loss‚1.6856923025870134 /loss/FRt2/detection_loss‚3.8730935537815094 1loss/SRqqt2/detection_loss‚2.4137844298333566 .loss/FBt1/detection_loss‚18.36475572465375 epoch‚0 % _timestamp‚1.7857942631697795e+09 /loss/FRt2/assignment_loss‚4.833236687978109 0loss/FBt1/assignment_loss‚0.6257114008601222 /loss/FRt1/detection_loss‚3.1150783057890687‘p¬øŒ‚ .loss/FBt1/detection_loss‚18.36475572465375 epoch‚0 % _timestamp‚1.7857942631697795e+09 /loss/FRt2/assignment_loss‚4.833236687978109 0loss/FBt1/assignment_loss‚0.6257114008601222 /loss/FRt1/detection_loss‚3.1150783057890687 1loss/SRqqt1/detection_loss‚1.7016665346373985 %loss/total_loss‚46.91370846942982 trainer/global_step‚2099 _runtime‚ 937.246829711 _step‚167 0loss/FRt1/assignment_loss‚3.4704618337107638 2loss/SRqqt2/assignment_loss‚1.7836387367584767 2loss/FBt2/assignment_loss‚0.061299103559576906 .loss/FBt2/detection_loss‚4.985289855280671 2loss/SRqqt1/assignment_loss‚1.6856923025870134 /loss/FRt2/detection_loss‚3.8730935537815094 1loss/SRqqt2/detection_loss‚2.4137844298333566§å謭œ‚(:‘ ÞÄÓ³ù¸ gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚117.481' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.16033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æòç=Ž‚(:²  ÞÄÓ’†Ñ¹ disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd2118.out‚0 disk.dm-0.in‚0$ proc.memory.rssMB‚11253.33984375 disk.nvme0n1.in‚0 network.sent‚ 2.0278e+06 disk.drbd1510.out‚0 disk.drbd1285.in‚0 disk.drbd1828.out‚0 disk.drbd2118.in‚0( disk./.usagePercent‚31.8956714554242% memory_percent‚41.212207078933716 disk.drbd1552.in‚0 disk.sda.in‚0 disk.drbd2041.out‚0 disk.drbd1673.out‚0 disk.drbd1285.out‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1931.out‚0 disk.sda2.out‚ 25.140625 disk.drbd1510.in‚0 disk.drbd1938.out‚0* proc.memory.availableMB‚38527.16796875 disk.rbd0.out‚0 disk.drbd1828.in‚0 proc.cpu.threads‚29 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1226.in‚0! disk.nvme0n1.out‚ 27.337890625 disk.drbd1356.in‚0 disk.drbd1550.out‚0 disk.drbd1931.in‚0 disk.sda2.in‚0 disk.drbd1552.out‚0 disk.drbd1965.out‚0 disk.rbd1.out‚ 0.46484375 disk.drbd2111.in‚0* proc.memory.percent‚17.171233892440796 disk.drbd1226.out‚0 disk.drbd2111.out‚0 disk.sda1.in‚0 network.recv‚181334 disk.rbd1.in‚0 disk.drbd2078.in‚0 disk.drbd1584.in‚0 disk.drbd1356.out‚0 disk.dm-0.out‚ 27.337890625 cpu‚14.065565760363569 disk.sda.out‚ 25.140625 disk.sda1.out‚0$ disk./.usageGB‚570.3269996643066 disk.drbd1673.in‚0 disk.drbd1817.out‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.rbd0.in‚0ðä 0+ _wandbruntime‚959  _runtime‚959îsªlÈ ' lr-AdamW/pg1‚5.5384428756922805e-05 ' lr-AdamW/pg2‚5.5384428756922805e-05 trainer/global_step‚2149 % _timestamp‚1.7857942851074128e+09 _runtime‚ 959.184408498 _step‚168€MuÍ‘Ç _runtime‚ 959.184408498 _step‚168 ' lr-AdamW/pg1‚5.5384428756922805e-05 ' lr-AdamW/pg2‚5.5384428756922805e-05 trainer/global_step‚2149 % _timestamp‚1.7857942851074128e+09¨ÿ:CÎ0’+ _wandbruntime‚959  _runtime‚959›b•¸'“²O UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36997420 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37001150358528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚37882715972736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22710391 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚38804600752128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14287027 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 120188416 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14629356 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38943853768192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14287027 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚38804496283136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6162411 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36987762 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚38804600752128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14287029 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1891430175232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38804496283136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22700735 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36913066107904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 145092096 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8466947 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36913083147264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1891517604864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35995932204832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14629608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35995915165472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1886886610388 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7857942851085374e+09 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 959.18577868 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1891517604864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6162661 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1301%mr½'”·O L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22700735 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36913066107904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 145092096 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8466947 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36913083147264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1891517604864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35995932204832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14629608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35995915165472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1886886610388 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7857942851085374e+09 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 959.18577868 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1891517604864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6162661 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36997420 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37001150358528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚37882715972736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22710391 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚38804600752128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14287027 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 120188416 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14629356 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38943853768192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14287027 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚38804496283136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6162411 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36987762 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚38804600752128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14287029 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1891430175232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38804496283136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0©… ] 0•+ _wandbruntime‚959  _runtime‚959¼kýþ&–¼M ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚36012672377152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚38822647141888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37014600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37014600 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_bóáY¡œatch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22720928 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚38962119405056 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37004928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 trainer/global_step‚2149 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1892309794816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6165252 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚38822542665728 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36930249910272 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8470882 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37018332571648 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36930232870912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 _runtime‚ 959.559978648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22720928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8470884 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1943631262720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1892397231616 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22711258 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36012655337792 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22711258 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14293670 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6165508 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1892309794816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14636392 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14636134 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1943761929728 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 $ _timestamp‚1.785794285482631e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0ã êŽÇ&—ÁM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22720928 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚38962119405056 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37004928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 trainer/global_step‚2149 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1892309794816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6165252 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚38822542665728 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36930249910272 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8470882 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37018332571648 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36930232870912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 _runtime‚ 959.559978648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22720928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8470884 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1943631262720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1892397231616 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22711258 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36012655337792 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22711258 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14293670 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6165508 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1892309794816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14636392 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14636134 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1943761929728 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 $ _timestamp‚1.785794285482631e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚36012672377152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚38822647141888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37014600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37014600 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200ªŸ{›á0˜+ _wandbruntime‚959  _runtime‚959¨·)œ™û /loss/FRt1/detection_loss‚3.0914352166594243 1loss/SRqqt1/detection_loss‚2.0969753909067745 1loss/SRqqt2/detection_loss‚2.3647815271051953 .loss/FBt2/detection_loss‚6.191679933798681 epoch‚0 2loss/SRqqt2/assignment_loss‚1.9683964521770019 /loss/FBt1/detection_loss‚13.299962868293127 trainer/global_step‚2149 % _timestamp‚1.7857942854844358e+09 _runtime‚ 959.561224966 _step‚171 1loss/FBt2/assignment_loss‚0.07612848471410782 /loss/FBt1/assignment_loss‚0.632962758342425 .loss/FRt2/detection_loss‚4.022613919207028 %loss/total_loss‚43.80985363730814 0loss/FRt1/assignment_loss‚3.6839042317633535 /loss/FRt2/assignment_loss‚4.813244155475071 2loss/SRqqt1/assignment_loss‚1.5677686988659527zZ¿¢š€ 0loss/FRt1/assignment_loss‚3.6839042317633535 /loss/FRt2/assignment_loss‚4.813244155475071 2loss/SRqqt1/assignment_loss‚1.5677686988659527 /loss/FRt1/detection_loss‚3.0914352166594243 1loss/SRqqt1/detection_loss‚2.0969753909067745 1loss/SRqqt2/detection_loss‚2.3647815271051953 .loss/FBt2/detection_loss‚6.191679933798681 epoch‚0 2loss/SRqqt2/assignment_loss‚1.9683964521770019 /loss/FBt1/detection_loss‚13.299962868293127 trainer/global_step‚2149 % _timestamp‚1.7857942854844358e+09 _runtime‚ 959.561224966 _step‚171 1loss/FBt2/assignment_loss‚0.07612848471410782 /loss/FBt1/assignment_loss‚0.632962758342425 .loss/FRt2/detection_loss‚4.022613919207028 %loss/total_loss‚43.80985363730814«¾tîf‘›‚(:† íÄÓ˜§•¹ gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚106.575' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚35.525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E>±]Aœ‚(:¶  íÄÓÂòÙ¹ disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.nvme0n1.in‚0 disk.drbd1965.in‚0 disk.drbd1550.out‚0 disk.drbd2111.in‚0 disk.drbd2078.out‚0 disk.rbd0.in‚0 disk.drbd1289.in‚0$ proc.memory.rssMB‚11253.33984375 disk.drbd2078.in‚0 disk.drbd1938.in‚0 disk.drbd1510.out‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0* proc.memory.percent‚17.171233892440796 disk.drbd1550.in‚0 network.recv‚183677 disk.rbd0.out‚0 disk.sda.out‚ 25.3203125 disk.drbd2041.in‚0 disk.sda1.in‚0 disk.drbd1510.in‚0! disk.nvme0n1.out‚ 27.474609375 disk.drbd1828.out‚0 disk.drbd1673.out‚0$ memory_percent‚41.21338725090027* proc.memory.availableMB‚38526.39453125 cpu‚14.058031376648005 proc.cpu.threads‚29 disk.drbd1584.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1938.out‚0 disk.drbd1673.in‚0 disk.drbd1817.out‚0 disk.rbd1.out‚ 0.46484375 disk.drbd1584.in‚0 disk.dm-0.in‚0 disk.drbd1552.in‚0$ disk./.usageGB‚570.3270416259766 disk.sda.in‚0 disk.drbd1817.in‚0 disk.drbd1356.out‚0 disk.dm-0.out‚ 27.474609375 disk.drbd1289.out‚0) disk./.usagePercent‚31.89567380214045 disk.sda2.in‚0 disk.drbd1226.in‚0 network.sent‚ 2.072687e+06 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.sda2.out‚ 25.3203125 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1965.out‚0 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1828.in‚0Ò¹¯w:‚(:¯  üÄÓúùÛ¹ disk.drbd1673.in‚0 disk.drbd1510.out‚0 disk.drbd1285.out‚0 disk.drbd1965.out‚0 disk.drbd1584.out‚0 disk.rbd0.out‚0 disk.drbd2111.in‚0% memory_percent‚41.212403774261475 disk.drbd1226.in‚0 disk.drbd1285.in‚0 disk.drbd1289.out‚0 proc.cpu.threads‚29 disk.sda.out‚25.625 network.recv‚186194 disk.sda1.out‚0 disk.drbd1931.in‚0! disk.nvme0n1.out‚ 27.755859375 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1552.in‚0 disk.rbd1.out‚ 0.46484375 disk.rbd0.in‚0 disk.drbd1510.in‚0 disk.drbd1356.out‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0) proc.memory.availableMB‚ 38527.0390625 disk.drbd2041.in‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd1550.out‚0 cpu‚14.051195704211692$ disk./.usageGB‚570.3270225524902 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚0 disk.drbd1289.in‚0 disk.drbd1673.out‚0 disk.drbd2118.out‚0 disk.dm-0.out‚ 27.755859375 disk.drbd1938.out‚0 disk.drbd1226.out‚0 disk.sda.in‚0 disk.dm-0.in‚0 disk.drbd1817.in‚0* proc.memory.percent‚17.171233892440796* disk./.usagePercent‚31.895672735451246 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd1552.out‚0 disk.sda2.out‚25.625 disk.drbd1828.in‚0 disk.drbd2118.in‚0$ proc.memory.rssMB‚11253.33984375 network.sent‚ 2.117617e+06SªÞ¨œž‚(:‘ üÄÓô‹éº gpu.0.gpu‚29.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚125.939' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.97966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|å€0Ÿ+ _wandbruntime‚980  _runtime‚980o-2$Ä ¾ % lr-AdamW/pg2‚5.66730380811881e-05 trainer/global_step‚2199 % _timestamp‚1.7857943068099031e+09 _runtime‚ 980.886879003 _step‚172 % lr-AdamW/pg1‚5.66730380811881e-05i }µÉ¡Ã trainer/global_step‚2199 % _timestamp‚1.7857943068099031e+09 _runtime‚ 980.886879003 _step‚172 % lr-AdamW/pg1‚5.66730380811881e-05 % lr-AdamW/pg2‚5.66730380811881e-05¬YœM 0¢+ _wandbruntime‚980  _runtime‚980jjŸ?¹'£³O JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 % _timestamp‚1.7857943068109653e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37771421297664 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37861506198528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8663797 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39706830568960 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1935426310656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1930689270772 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14619179 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚39706935037952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1935426310656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚36832940820832 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚39706935037952 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37771421297664 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1988050531840 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚36832923781472 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14619177 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1930775073896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1935513740288 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39849556730368 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚38763613052244 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23238393 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23228737 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23238393 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14969692 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6306138 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 _runtime‚ 980.888122625 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1935513740288 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 trainer/global_step‚2199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ù¬}÷¯¤¸O GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8663797 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39706830568960 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1935426310656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1930689270772 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14619179 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚39706935037952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1935426310656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚36832940820832 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚39706935037952 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37771421297664 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1988050531840 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚36832923781472 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14619177 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1930775073896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1935513740288 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39849556730368 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚38763613052244 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23238393 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23228737 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23238393 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14969692 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6306138 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 _runtime‚ 980.888122625 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1935513740288 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 trainer/global_step‚2199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 % _timestamp‚1.7857943068109653e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37771421297664 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37861506198528­í¶l0¥+ _wandbruntime‚981  _runtime‚981• ×Ã&¦½M UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8667732 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39867668739072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1988812173824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37874752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 143249920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23239260 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚36849680993152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23248930 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23239260 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1936393367040 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14625820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37865080 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚38781333540382 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14976696 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚37788588060672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39867500585472 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚38781230697786 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14976450 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚37788571021312 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39724981427712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6308962 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚2199 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1931566743994 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1936305930240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168153600 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7857943071820307e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1936305930240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14625822 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14625822 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23248930 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _runtime‚ 981.259350024 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37865080 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024s/ÇÈ&§ÂM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37878688411648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚39724876951552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚37788588060672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6308718 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1931652547230 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚38781333540382 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14976696 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚37788588060672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39867500585472 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚38781230697786 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14976450 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚37788571021312 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39724981427712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6308962 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚2199 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1931566743994 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1936305930240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168153600 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7857943071820307e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1936305930240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14625822 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14625822 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23248930 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _runtime‚ 981.259350024 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37865080 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8667732 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39867668739072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1988812173824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37874752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 143249920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23239260 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚36849680993152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23248930 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23239260 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1936393367040 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14625820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37865080 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 EÙs2­- _wandbruntime‚1003  _runtime‚1003܎ɮà ' lr-AdamW/pg1‚5.7961647405453414e-05 ' lr-AdamW/pg2‚5.7961647405453414e-05 trainer/global_step‚2249 % _timestamp‚1.7857943291883054e+09 _runtime‚1003.265362029 _step‚176ÎsýVÎ¯È _runtime‚1003.265362029 _step‚176 ' lr-AdamW/pg1‚5.7961647405453414e-05 ' lr-AdamW/pg2‚5.7961647405453414e-05 trainer/global_step‚2249 % _timestamp‚1.7857943291883054e+09°‡*•2°- _wandbruntime‚1003  _runtime‚1003äùdº'±´O WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1979422446080 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1979422446080 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23756739 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1979509875712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23766395 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2033069776896 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8860647 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38708066 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚38629759448064 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15310648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14951329 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40609164854784 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23756739 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 _step‚177 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14951329 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1974577734280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚39644612974236 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38717724 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15310409 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚38629759448064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚40755062489600 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6449764 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚38721862038528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8860645 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚39644510131752 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23766395 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40609164854784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2249 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚38629742408704 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6450001 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _runtime‚1003.266779295'`Ÿî¿'²¹O WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8860647 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38708066 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚38629759448064 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15310648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14951329 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40609164854784 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23756739 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 _step‚177 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14951329 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1974577734280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚39644612974236 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38717724 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15310409 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚38629759448064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚40755062489600 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6449764 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚38721862038528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8860645 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚39644510131752 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23766395 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40609164854784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2249 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚38629742408704 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6450001 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _runtime‚1003.266779295 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1979422446080 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1979422446080 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23756739 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1979509875712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23766395 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2033069776896 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121±óa³ß2³- _wandbruntime‚1003  _runtime‚10032Å7£Ã&´½M SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14957972 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2033968851456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15317183 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _step‚178 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚40627211237376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1980389502464 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15317459 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2034105809920 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38734904 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14957970 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚40773013103104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39662230619890 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23776932 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14957970 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1980302065664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚38646909171712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38734904 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚38739069155328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚38646909171712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23776932 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14957970 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1980302065664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚38646909171712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38734904 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚38739069155328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚38646909171712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚ï‚ÐÍ 130 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15317183 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _step‚178 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚40627211237376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1980389502464 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15317459 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2034105809920 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38734904 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14957970 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚40773013103104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39662230619890 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640²oø2¶- _wandbruntime‚1003  _runtime‚1003™­ƒ·ú 0loss/FRt1/assignment_loss‚3.6879856789365726 1loss/SRqqt1/assignment_loss‚1.496702172842465 1loss/SRqqt2/detection_loss‚2.9354374638437815 %loss/total_loss‚65.77386626517975 epoch‚0 trainer/global_step‚2249 % _timestamp‚1.7857943295733893e+09 _runtime‚1003.650180542 /loss/FRt1/detection_loss‚3.2566837492458363 _step‚179 /loss/FRt2/assignment_loss‚4.635836203893025 2loss/SRqqt2/assignment_loss‚2.0248920209165933 1loss/FBt1/assignment_loss‚0.46823592769214883 .loss/FRt2/detection_loss‚4.011511661868164 /loss/SRqqt1/detection_loss‚1.97434836009358 .loss/FBt1/detection_loss‚36.02511526147524 .loss/FBt2/detection_loss‚5.220499950824735 1loss/FBt2/assignment_loss‚0.03661781354759146®‰Az¸ÿ 0loss/FRt1/assignment_loss‚3.6879856789365726 1loss/SRqqt1/assignment_loss‚1.496702172842465 1loss/SRqqt2/detection_loss‚2.9354374638437815 %loss/total_loss‚65.77386626517975 epoch‚0 trainer/global_step‚2249 % _timestamp‚1.7857943295733893e+09 _runtime‚1003.650180542 /loss/FRt1/detection_loss‚3.2566837492458363 _step‚179 /loss/FRt2/assignment_loss‚4.635836203893025 2loss/SRqqt2/assignment_loss‚2.0248920209165933 1loss/FBt1/assignment_loss‚0.46823592769214883 .loss/FRt2/detection_loss‚4.011511661868164 /loss/SRqqt1/detection_loss‚1.97434836009358 .loss/FBt1/detection_loss‚36.02511526147524 .loss/FBt2/detection_loss‚5.220499950824735 1loss/FBt2/assignment_loss‚0.03661781354759146³¶"kêF¹‚(:»  šžÄÓÿÆÒ¹ disk.dm-0.out‚28.25146484375 disk.drbd1356.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.sda.in‚0 disk.sda.out‚ 26.23828125 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd1289.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 network.sent‚ 2.165487e+06 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0# memory_percent‚41.2122368812561 disk.drbd1938.out‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.rbd0.in‚0 disk.sda2.out‚ 26.23828125$ proc.memory.rssMB‚11253.33984375 disk.drbd2078.out‚0) proc.memory.availableMB‚ 38527.1484375 network.recv‚189338 disk.dm-0.in‚0 disk.drbd1673.in‚0 disk.drbd1510.out‚0 disk.drbd1285.out‚0* disk./.usagePercent‚31.895674015478292 disk.drbd1550.in‚0 disk.drbd1356.in‚0 disk.drbd1965.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd2118.in‚0 disk.sda1.in‚0# disk.nvme0n1.out‚28.25146484375 disk.rbd0.out‚0 proc.cpu.threads‚29 disk.drbd1226.in‚0 disk.rbd1.in‚0 cpu‚14.037168089709322 disk.drbd2041.out‚0 disk.drbd1673.out‚0* proc.memory.percent‚17.171233892440796 disk.drbd1817.out‚0 disk.nvme0n1.in‚0 disk.rbd1.out‚ 0.46484375$ disk./.usageGB‚570.3270454406738 disk.drbd1550.out‚0 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd1584.out‚0 disk.drbd1226.out‚0 disk.drbd1285.in‚0 disk.sda2.in‚0 disk.drbd1828.out‚0›Ê§ºœº‚(:‘ šžÄÓÝ·‹º gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚123.559' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.18633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}¨$í›»‚(: ©žÄÓñ¶’¹ gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚97.589' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚32.52966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~'=‡>¼‚(:³  ©žÄÓªÛй disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1931.in‚0 disk.dm-0.out‚28.47021484375 disk.sda2.out‚26.625 disk.drbd2041.out‚0 disk.drbd2041.in‚0* proc.memory.percent‚17.171233892440796 disk.dm-0.in‚0 disk.drbd1356.out‚0 disk.drbd1938.out‚0$ proc.memory.rssMB‚11253.33984375 cpu‚14.030559165153639$ memory_percent‚41.21307730674744 proc.cpu.threads‚29 network.recv‚191729 disk.sda2.in‚0 disk.drbd1510.out‚0 disk.drbd1285.in‚0 disk.drbd1965.out‚0 disk.rbd1.out‚ 0.46484375 disk.drbd2118.out‚0 disk.drbd1226.out‚0 disk.drbd2111.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0* disk./.usagePercent‚31.895672948789088$ disk./.usageGB‚570.3270263671875 disk.drbd1673.out‚0 disk.nvme0n1.in‚0 disk.rbd0.out‚0# disk.nvme0n1.out‚28.47021484375 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1289.in‚0 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd1552.in‚0 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.sda1.in‚0 disk.drbd1289.out‚0 network.sent‚ 2.210542e+06 disk.drbd1817.in‚0 disk.drbd1510.in‚0 disk.sda.in‚0* proc.memory.availableMB‚38526.59765625 disk.drbd1938.in‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.sda.out‚26.625 disk.drbd1552.out‚0 disk.rbd0.in‚0 disk.drbd1673.in‚0 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.drbd1965.in‚0¹ÔP2½- _wandbruntime‚1025  _runtime‚10254x1‹É¾Ã ' lr-AdamW/pg1‚5.9250256729718725e-05 ' lr-AdamW/pg2‚5.9250256729718725e-05 trainer/global_step‚2299 % _timestamp‚1.7857943511546633e+09 _runtime‚1025.231633373 _step‚180ÍY¡ÝÎ¿È trainer/global_step‚2299 % _timestamp‚1.7857943511546633e+09 _runtime‚1025.231633373 _step‚180 ' lr-AdamW/pg1‚5.9250256729718725e-05 ' lr-AdamW/pg2‚5.9250256729718725e-05´iTES2À- _wandbruntime‚1025  _runtime‚1025—ƒ¬ º'Á´O ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2023506011136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6594300 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚39488097598464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39488080559104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15283479 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2078414438400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9057495 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24294397 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚222 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41660472544768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6594522 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2023418581504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 % _timestamp‚1.7857943511557093e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚39488080559104 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39577876 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2078279569920 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40525407211260 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _step‚181 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41511603609600 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2023418581504 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚40525510053744 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2023506011136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24284741 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2018552000912 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚39582217878528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚38506941013472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15283479 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41511499140608 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2018466197788 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41511499140608 trainer/global_step‚2299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9057497 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚41660632316928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39582192974848 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24284741 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚38506958052832 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15283477 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚39488097598464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24294397 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6594522 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2023418581504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 % _timestamp‚1.7857943511557093e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚39488080559104 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39577876 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2078279569920 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40525407211260 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _step‚181 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41511603609600 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2023418581504 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚40525510053744 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2023506011136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24284741 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2018552000912 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚39582217878528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚38506941013472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15283479 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41511499140608 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2018466197788 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41511499140608 trainer/global_step‚2299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9057497 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚41660632316928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39582192974848 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24284741 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚38506958052832 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15283477 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚39488097598464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24294397 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9057495 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24294397 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚222 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41660472544768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887µ#Ü2Ã- _wandbruntime‚1025  _runtime‚1025ŸÉÁoÄ&ľM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2019343671010 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39595056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2079203138560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚1025.605866985 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39505247322112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _step‚182 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38523681185792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39585384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15290122 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41678754606592 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15290120 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24295264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2019429474246 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39585384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39505247322112 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2024385637888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39595056 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41529649999360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24295264 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15658795 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39505264361472 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15658547 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9061432 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6597115 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15290120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41529545523200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24304934 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39599424995328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2024298201088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 trainer/global_step‚2299 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6597361 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9061434 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39599400091648 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚41529649999360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24304934 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40543127699398 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚38523698225152 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39505264361472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2024385637888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15290122 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2024298201088 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40543024856802ÛùyÉ&ÅÃM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39505247322112 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2024385637888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39595056 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41529649999360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24295264 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15658795 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39505264361472 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15658547 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9061432 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6597115 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15290120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41529545523200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24304934 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39599424995328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2024298201088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 trainer/global_step‚2299 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6597361 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9061434 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39599400091648 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚41529649999360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24304934 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40543127699398 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚38523698225152 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39505264361472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2024385637888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15290122 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2024298201088 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40543024856802 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2019343671010 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39595056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2079203138560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚1025.605866985 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39505247322112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _step‚182 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38523681185792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39585384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15290122 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41678754606592 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15290120 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24295264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2019429474246 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39585384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121¶ºwÆ2Æ- _wandbruntime‚1025  _runtime‚1025Fõ¯þÇø 1loss/SRqqt1/detection_loss‚1.6468438944214838 $loss/total_loss‚50.6128105173447 2loss/SRqqt1/assignment_loss‚1.4926543120448863 _runtime‚1025.607263933 _step‚183 0loss/FRt1/assignment_loss‚3.3963332148485406 0loss/FBt2/assignment_loss‚0.1553832309201915 .loss/FBt2/detection_loss‚5.914044667690184 epoch‚0 $ _timestamp‚1.785794351530451e+09 /loss/FRt2/assignment_loss‚4.744643178853122 2loss/SRqqt2/assignment_loss‚1.8959436613908955 0loss/FBt1/assignment_loss‚1.0032909636585308 .loss/FRt1/detection_loss‚3.616229638803837 .loss/FRt2/detection_loss‚4.101250869758202 0loss/SRqqt2/detection_loss‚3.104426113706419 /loss/FBt1/detection_loss‚19.541766771248408 trainer/global_step‚2299N-†Èý trainer/global_step‚2299 1loss/SRqqt1/detection_loss‚1.6468438944214838 $loss/total_loss‚50.6128105173447 2loss/SRqqt1/assignment_loss‚1.4926543120448863 _runtime‚1025.607263933 _step‚183 0loss/FRt1/assignment_loss‚3.3963332148485406 0loss/FBt2/assignment_loss‚0.1553832309201915 .loss/FBt2/detection_loss‚5.914044667690184 epoch‚0 $ _timestamp‚1.785794351530451e+09 /loss/FRt2/assignment_loss‚4.744643178853122 2loss/SRqqt2/assignment_loss‚1.8959436613908955 0loss/FBt1/assignment_loss‚1.0032909636585308 .loss/FRt1/detection_loss‚3.616229638803837 .loss/FRt2/detection_loss‚4.101250869758202 0loss/SRqqt2/detection_loss‚3.104426113706419 /loss/FBt1/detection_loss‚19.541766771248408·|†¡Æ‘É‚(:† ¸žÄÓé…ÿ¸ gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚132.216' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚44.072 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0güÓ’9Ê‚(:®  ¸žÄӻŹ network.sent‚ 2.213534e+06 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1285.out‚0 disk.sda2.out‚26.9375 disk.drbd1550.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1673.out‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1965.in‚0 disk.sda2.in‚0* disk./.usagePercent‚31.895672948789088 disk.drbd2118.in‚0 disk.drbd1510.in‚0$ disk./.usageGB‚570.3270263671875 network.recv‚192802 disk.nvme0n1.in‚0* proc.memory.percent‚17.171233892440796 disk.drbd2041.in‚0 disk.drbd1931.in‚0! disk.nvme0n1.out‚ 28.802734375 disk.rbd0.in‚0 disk.drbd1289.in‚0 disk.drbd1584.in‚0 disk.dm-0.out‚ 28.802734375 disk.drbd1938.out‚0 disk.drbd1226.out‚0& proc.memory.availableMB‚ 38526.5625$ proc.memory.rssMB‚11253.33984375 disk.drbd1552.in‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.sda.out‚26.9375 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.drbd1931.out‚0 disk.drbd1817.in‚0 disk.drbd1552.out‚0 disk.rbd1.out‚ 0.46484375 disk.rbd0.out‚0 disk.drbd2041.out‚0 disk.dm-0.in‚0 disk.drbd2078.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.drbd1285.in‚0 cpu‚14.024359175903653 proc.cpu.threads‚29 disk.rbd1.in‚0% memory_percent‚41.213130950927734ãûœ<2Ë- _wandbruntime‚1047  _runtime‚1047öí ¿ÇÌÁ & lr-AdamW/pg1‚6.053886605398402e-05 & lr-AdamW/pg2‚6.053886605398402e-05 trainer/global_step‚2349 % _timestamp‚1.7857943733613405e+09 _runtime‚1047.438313803 _step‚184RÄÌÍÆ trainer/global_step‚2349 % _timestamp‚1.7857943733613405e+09 _runtime‚1047.438313803 _step‚184 & lr-AdamW/pg1‚6.053886605398402e-05 & lr-AdamW/pg2‚6.053886605398402e-05¸b7ã2Î- _wandbruntime‚1047  _runtime‚1047-‹c”Ï´O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15615627 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2062440464420 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2123707158016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2067414716928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2067414716928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24822399 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40442548814848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚2349 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40438028 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15615629 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40346418709504 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24822399 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42413937895424 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚40346435748864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9254345 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40442573718528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42413833426432 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2062354661296 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚42413937895424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40428370 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15615627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2067502146560 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2067502146560 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24812743 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6736156 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚42566129492992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 _runtime‚1047.442609943 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42413833426432 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚42566280876544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41406304290768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚40346418709504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peakÕ¾V&‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 % _timestamp‚1.7857943733624022e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24812743 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24822399 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40442548814848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚2349 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40438028 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15615629 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40346418709504 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24822399 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42413937895424 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚40346435748864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9254345 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40442573718528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42413833426432 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2062354661296 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚42413937895424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40428370 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15615627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2067502146560 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2067502146560 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24812743 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6736156 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚42566129492992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 _runtime‚1047.442609943 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42413833426432 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚42566280876544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41406304290768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚40346418709504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 % _timestamp‚1.7857943733624022e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24812743 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15615627 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2062440464420 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2123707158016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2067414716928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0¹çÀ2Ñ- _wandbruntime‚1047  _runtime‚1047 kíÄ&Ò¾M YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2068381773312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15997542 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚42584401978368 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40459755931648 _runtime‚1047.803865712 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42431879809024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15997287 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚41423921936310 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9258284 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39360689801792 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24832936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15622272 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40459780835328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 % _timestamp‚1.7857943737265382e+09 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40455208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15622272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15622270 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚40363602511872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚186 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚40363585472512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6739258 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40445536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 trainer/global_step‚2349 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24823266 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42431984285184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚39360706841152 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9258282 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚42584244310528 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40363602511872 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9258284 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39360689801792 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24832936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15622272 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40459780835328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 % _timestamp‚1.7857943737265382e+09 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40455208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15622272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15622270 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚40363602511872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚186 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚40363585472512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6739258 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40445536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 trainer/global_step‚2349 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24823266 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42431984285184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚39360706841152 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9258282 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚42584244310528 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40363602511872 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42431879809024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15997287 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚41423921936310º{‹®2Ô- _wandbruntime‚1047  _runtime‚1047)öüÕö -loss/FBt1/detection_loss‚19.6685317328998 2loss/SRqqt1/assignment_loss‚1.6751401164382695 .loss/FRt2/detection_loss‚3.806560022310472 1loss/SRqqt1/detection_loss‚1.9310318533730606 epoch‚0 _step‚187 /loss/FRt1/assignment_loss‚3.357873094979153 0loss/FBt1/assignment_loss‚0.7396863832816182 -loss/FBt2/detection_loss‚7.88662684222668 $loss/total_loss‚50.9204624309435 % _timestamp‚1.7857943737283556e+09 /loss/FRt2/assignment_loss‚4.697920406368417 1loss/FBt2/assignment_loss‚0.03161547036996063 /loss/FRt1/detection_loss‚3.1978488013308537 0loss/SRqqt2/detection_loss‚2.028826130699748 trainer/global_step‚2349 _runtime‚ 1047.80516145 2loss/SRqqt2/assignment_loss‚1.8988015766654696;]:ÿÖû 2loss/SRqqt1/assignment_loss‚1.6751401164382695 .loss/FRt2/detection_loss‚3.806560022310472 1loss/SRqqt1/detection_loss‚1.9310318533730606 epoch‚0 _step‚187 /loss/FRt1/assignment_loss‚3.357873094979153 0loss/FBt1/assignment_loss‚0.7396863832816182 -loss/FBt2/detection_loss‚7.88662684222668 $loss/total_loss‚50.9204624309435 % _timestamp‚1.7857943737283556e+09 /loss/FRt2/assignment_loss‚4.697920406368417 1loss/FBt2/assignment_loss‚0.03161547036996063 /loss/FRt1/detection_loss‚3.1978488013308537 0loss/SRqqt2/detection_loss‚2.028826130699748 trainer/global_step‚2349 _runtime‚ 1047.80516145 2loss/SRqqt2/assignment_loss‚1.8988015766654696 -loss/FBt1/detection_loss‚19.6685317328998»kœµÙœ×‚(:‘ ÇžÄÓ£õ‘¹ gpu.0.gpu‚24.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚100.342' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚33.44733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lø½S;Ø‚(:°  ÇžÄÓêßع cpu‚14.017200018027413 disk.drbd1931.in‚0 disk.rbd0.out‚0 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.drbd2078.in‚0 disk.drbd1550.in‚0 disk.sda.in‚0* proc.memory.availableMB‚38526.48828125 disk.rbd1.out‚ 0.46484375 disk.drbd1828.in‚0 disk.drbd1828.out‚0* proc.memory.percent‚17.171233892440796 disk.drbd1226.out‚0! disk.nvme0n1.out‚ 28.990234375 network.recv‚195348 disk.drbd1938.out‚0 disk.drbd1226.in‚0 disk.drbd1285.in‚0 disk.drbd1965.out‚0 disk.drbd1584.out‚0$ memory_percent‚41.21324419975281 disk.drbd2041.out‚0 network.sent‚ 2.258415e+06 disk.drbd1552.in‚0 disk.drbd1965.in‚0 disk.rbd0.in‚0 disk.drbd2111.out‚0 proc.cpu.threads‚29 disk.nvme0n1.in‚0 disk.rbd1.in‚0 disk.drbd2078.out‚0 disk.dm-0.out‚ 28.990234375 disk.drbd1673.out‚0 disk.sda2.out‚27.1875) disk./.usagePercent‚31.89570153605976 disk.sda1.in‚0$ proc.memory.rssMB‚11253.33984375$ disk./.usageGB‚570.3275375366211 disk.sda2.in‚0 disk.drbd1584.in‚0 disk.sda1.out‚0 disk.drbd1938.in‚0 disk.drbd1817.in‚0 disk.drbd1285.out‚0 disk.drbd2111.in‚0 disk.drbd1673.in‚0 disk.drbd1510.out‚0 disk.dm-0.in‚0 disk.drbd1289.out‚0 disk.sda.out‚27.1875 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.drbd1817.out‚0 disk.drbd1510.in‚0 disk.drbd1356.in‚0 disk.drbd1931.out‚0 disk.drbd1552.out‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0÷ q´œÙ‚(:‘ ÖžÄÓµ‘è¸ gpu.0.gpu‚22.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚124.901' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.63366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥í%‘DÚ‚(:¹  ÖžÄÓ×ʪ¹ disk.dm-0.out‚ 29.5166015625 disk.drbd1552.out‚0 disk.sda2.in‚0 disk.drbd2111.in‚0* proc.memory.availableMB‚38527.12109375 disk.drbd2041.out‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1550.out‚0 disk.sda.in‚0$ disk./.usageGB‚570.3275184631348 disk.drbd1938.out‚0 cpu‚14.01156920841747 disk.sda2.out‚ 27.83203125 disk.drbd2041.in‚0 disk.drbd1931.in‚0 disk.drbd1817.in‚0 disk.drbd1510.out‚0" disk.nvme0n1.out‚ 29.5166015625 disk.drbd1285.in‚0) disk./.usagePercent‚31.89570046937056 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0% memory_percent‚41.212278604507446 network.recv‚197708 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.rbd0.out‚0 disk.drbd2078.in‚0 disk.drbd1584.in‚0 proc.cpu.threads‚29 disk.drbd2118.out‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.dm-0.in‚0 disk.drbd1673.in‚0 disk.drbd1510.in‚0 disk.drbd1965.in‚0$ proc.memory.rssMB‚11253.33984375 disk.drbd1817.out‚0 disk.nvme0n1.in‚0 disk.drbd1356.in‚0 disk.drbd1552.in‚0* proc.memory.percent‚17.171233892440796 disk.drbd2118.in‚0 disk.sda.out‚ 27.83203125 disk.drbd1938.in‚0 disk.drbd1285.out‚0 disk.drbd1356.out‚0 network.sent‚ 2.30354e+06 disk.rbd0.in‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 0.46484375"‘Œ2Û- _wandbruntime‚1069  _runtime‚1069¥ÃPÆÜÀ & lr-AdamW/pg1‚6.182747537824933e-05 & lr-AdamW/pg2‚6.182747537824933e-05 trainer/global_step‚2399 $ _timestamp‚1.785794395625878e+09 _runtime‚1069.702826463 _step‚188e|AÌËÝÅ & lr-AdamW/pg2‚6.182747537824933e-05 trainer/global_step‚2399 $ _timestamp‚1.785794395625878e+09 _runtime‚1069.702826463 _step‚188 & lr-AdamW/pg1‚6.182747537824933e-05¼»:S2Þ- _wandbruntime‚1069  _runtime‚1069HùÊ|º'ß´O LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41302929558528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚42287304212760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41298180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚41204773899264 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚43316167712256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2111498281984 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15947779 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚41204756859904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15947777 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚41204773899264 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6880494 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚40180975284832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25340745 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2111410852352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚43316167712256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6880749 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25350401 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15947777 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ˆ K}¿'à¹O E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41288522 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚42287201370276 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚41204756859904 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25340745 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41302904654848 _runtime‚1069.704108508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2168783853056 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15947779 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43471844085760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2168914527232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25350401 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚43316272181248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16331946 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15947777 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41302929558528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚42287304212760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41298180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚41204773899264 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚43316167712256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2111498281984 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15947779 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚41204756859904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15947777 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚41204773899264 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6880494 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚40180975284832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25340745 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2111410852352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚43316167712256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6880749 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25350401 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2169815552512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41315360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚41221923622912 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41315360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43489952227840 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚43334318571008 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚43334318571008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43489794560000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16338502 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9455132 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 % _timestamp‚1.7857943959903555e+09 _step‚190 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚41221940662272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚42304819015818 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2169682788352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2112377908736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41305688 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2107206401262 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚40197698417792 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25351268 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15954420 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6883370 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚40197715457152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15954422 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25351268 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41305688 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15954422 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25360938 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16338753 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15954420 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2112377908736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚43334214094848 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2112290471936 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9455134 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887Åí¿É&ãÃM REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41315360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43489952227840 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚43334318571008 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚43334318571008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43489794560000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16338502 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9455132 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 % _timestamp‚1.7857943959903555e+09 _step‚190 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚41221940662272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚42304819015818 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2169682788352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2112377908736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41305688 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2107206401262 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚40197698417792 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25351268 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15954420 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6883370 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚40197715457152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15954422 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25351268 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41305688 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15954422 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25360938 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16338753 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15954420 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2112377908736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚43334214094848 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2112290471936 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9455134 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚41221940662272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚41221923622912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2112290471936 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚42304921858414 trainer/global_step‚2399 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2169815552512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41315360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚41221923622912¾­ B2ä- _wandbruntime‚1070  _runtime‚1070Ô·åû 2loss/SRqqt1/assignment_loss‚1.4584918191272116 .loss/FBt1/detection_loss‚8.977805334058674 epoch‚0 trainer/global_step‚2399 /loss/FRt2/assignment_loss‚4.818055859531265 2loss/SRqqt2/assignment_loss‚2.1243800980145813 1loss/FBt1/assignment_loss‚0.26249377548017283 0loss/SRqqt1/detection_loss‚1.908785990457834 .loss/FBt2/detection_loss‚3.370197344314165 _step‚191 2loss/FBt2/assignment_loss‚0.026376893645455743 0loss/SRqqt2/detection_loss‚3.477079754326318 %loss/total_loss‚36.28213536609057 _runtime‚1070.068911469 /loss/FRt1/assignment_loss‚3.565312003602787 /loss/FRt1/detection_loss‚2.9604491415832723 /loss/FRt2/detection_loss‚3.3327073519488417 $ _timestamp‚1.785794395992148e+09-#ùæ€ _step‚191 2loss/FBt2/assignment_loss‚0.026376893645455743 0loss/SRqqt2/detection_loss‚3.477079754326318 %loss/total_loss‚36.28213536609057 _runtime‚1070.068911469 /loss/FRt1/assignment_loss‚3.565312003602787 /loss/FRt1/detection_loss‚2.9604491415832723 /loss/FRt2/detection_loss‚3.3327073519488417 $ _timestamp‚1.785794395992148e+09 2loss/SRqqt1/assignment_loss‚1.4584918191272116 .loss/FBt1/detection_loss‚8.977805334058674 epoch‚0 trainer/global_step‚2399 /loss/FRt2/assignment_loss‚4.818055859531265 2loss/SRqqt2/assignment_loss‚2.1243800980145813 1loss/FBt1/assignment_loss‚0.26249377548017283 0loss/SRqqt1/detection_loss‚1.908785990457834 .loss/FBt2/detection_loss‚3.370197344314165¿¡ù¾[œç‚(:‘ åžÄÓ¡è¹ gpu.0.gpu‚29.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚124.418' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.47266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŧê?è‚(:´  åžÄÓܧî¹ disk.dm-0.in‚0 disk.drbd1552.out‚0 network.sent‚ 2.306542e+06 disk.drbd2118.in‚0 disk.drbd1938.in‚0" disk.nvme0n1.out‚ 29.7158203125 disk.drbd1552.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚28.09375 disk.drbd2041.out‚0 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.drbd1965.out‚0 disk.drbd1289.out‚0 disk.drbd1550.out‚0 proc.cpu.threads‚29$ disk./.usageGB‚570.3275184631348 disk.drbd1584.out‚0 disk.sda1.in‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 cpu‚14.005845057769418 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1673.in‚0 disk.nvme0n1.in‚0 network.recv‚198708 disk.rbd1.out‚ 0.46484375$ memory_percent‚41.21273159980774 disk.dm-0.out‚ 29.7158203125 disk.drbd1356.out‚0 disk.drbd1550.in‚0 disk.drbd1510.in‚0 disk.drbd1356.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda.out‚28.09375 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd2078.out‚0* proc.memory.availableMB‚38526.82421875 disk.drbd2111.out‚0 disk.drbd1289.in‚0$ proc.memory.rssMB‚11253.33984375* proc.memory.percent‚17.171233892440796 disk.drbd1828.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0) disk./.usagePercent‚31.89570046937056 disk.drbd1584.in‚0 disk.drbd1226.out‚0 disk.drbd1285.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0ÁÕlc2é- _wandbruntime‚1091  _runtime‚1091/çÍ$ÇêÁ & lr-AdamW/pg2‚6.311608470251464e-05 trainer/global_step‚2449 % _timestamp‚1.7857944175589917e+09 _runtime‚1091.635949993 _step‚192 & lr-AdamW/pg1‚6.311608470251464e-05oÆJôÌëÆ % _timestamp‚1.7857944175589917e+09 _runtime‚1091.635949993 _step‚192 & lr-AdamW/pg1‚6.311608470251464e-05 & lr-AdamW/pg2‚6.311608470251464e-05 trainer/global_step‚2449À]LǼ2ì- _wandbruntime‚1091  _runtime‚1091oK;º'í´O _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2213989336064 trainer/global_step‚2449 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚43168098449784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2155406987776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25878403 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16279929 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42063112049664 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚42163260494848 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7023892 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16672185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16279927 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚44218606467072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44218606467072 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16671937 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41017983900832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16279929 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44218501998080 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16279927 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42148674 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2150131588312 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25868747 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 % _timestamp‚1.7857944175600176e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25878403 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42063095010304 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25868747 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43168201292268 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44218501998080 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2155494417408 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9648045 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚44377411700224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2155406987776 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42063112049664 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚42163285398528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41017966861472 _step‚193 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42158332.è!–î¹O PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16279929 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42063112049664 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚42163260494848 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7023892 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16672185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16279927 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚44218606467072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 JDDeviceStatsMonitor.on_train_batch_start/oversZ«nŒ)ize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44218606467072 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16671937 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41017983900832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16279929 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44218501998080 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16279927 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42148674 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2150131588312 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25868747 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 % _timestamp‚1.7857944175600176e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25878403 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42063095010304 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25868747 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43168201292268 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44218501998080 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2155494417408 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9648045 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚44377411700224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2155406987776 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42063112049664 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚42163285398528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41017966861472 _step‚193 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42158332 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2213989336064 trainer/global_step‚2449 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚43168098449784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2155406987776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25878403 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121ÁY'þÀ2ï- _wandbruntime‚1092  _runtime‚1092䣼Ã&ð½M QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9651982 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2151094864770 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚43185716095326 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42080278812672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41034724073152 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44236548380672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7026745 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25888940 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1092.003716204 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16286572 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16678727 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2215026173952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16286572 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2156286607360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16678988 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9651984 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚44395358924288 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42165840 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚44395518689280 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44236652856832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚42180492515328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2214891312640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2449 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2156374044160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42080278812672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42175512 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 $ _timestamp‚1.785794417926383e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42175512 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 $ _timestamp‚1.785794417926383e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42080278812672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41034724073152 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44236548380672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7026745 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25888940 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1092.003716204 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16286572 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16678727 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2215026173952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16286572 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2156286607360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16678988 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9651984 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚44395358924288 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42165840 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚44395518689280 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44236652856832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚42180492515328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2214891312640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¦ &2ò- _wandbruntime‚1092  _runtime‚10922«óû 1loss/FBt2/assignment_loss‚0.28381578029883614 0loss/SRqqt2/detection_loss‚3.395386353190666 /loss/FBt1/detection_loss‚10.632133230566978 &loss/total_loss‚38.830495892088194 % _timestamp‚1.7857944179281135e+09 0loss/FRt1/assignment_loss‚3.8468272491943005 1loss/SRqqt1/assignment_loss‚1.491101997538849 1loss/SRqqt1/detection_loss‚1.5192942737153283 trainer/global_step‚2449 _runtime‚1092.004878323 /loss/FRt2/assignment_loss‚4.627165752980444 2loss/SRqqt2/assignment_loss‚1.9174888590498576 epoch‚0 _step‚195 /loss/FBt1/assignment_loss‚0.577022536584991 /loss/FRt1/detection_loss‚3.3791718853767527 .loss/FRt2/detection_loss‚3.727743897587061 .loss/FBt2/detection_loss‚3.433344076004131Çó=Œô€ /loss/FRt1/detection_loss‚3.3791718853767527 .loss/FRt2/detection_loss‚3.727743897587061 .loss/FBt2/detection_loss‚3.433344076004131 1loss/FBt2/assignment_loss‚0.28381578029883614 0loss/SRqqt2/detection_loss‚3.395386353190666 /loss/FBt1/detection_loss‚10.632133230566978 &loss/total_loss‚38.830495892088194 % _timestamp‚1.7857944179281135e+09 0loss/FRt1/assignment_loss‚3.8468272491943005 1loss/SRqqt1/assignment_loss‚1.491101997538849 1loss/SRqqt1/detection_loss‚1.5192942737153283 trainer/global_step‚2449 _runtime‚1092.004878323 /loss/FRt2/assignment_loss‚4.627165752980444 2loss/SRqqt2/assignment_loss‚1.9174888590498576 epoch‚0 _step‚195 /loss/FBt1/assignment_loss‚0.577022536584991ÃÔè¶æõ‚(:… ôžÄÓ°ô¸ gpu.0.gpu‚4.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚124.098' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚41.366 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á…ÄqCö‚(:¸  ôžÄÓ¸²´¹ disk.drbd1550.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.dm-0.in‚0 disk.drbd1938.out‚0 disk.drbd1817.out‚0 disk.drbd1285.out‚0 disk.drbd2078.out‚0 disk.drbd1510.in‚0 disk.nvme0n1.in‚0 disk.drbd2111.out‚0 disk.drbd2118.in‚0 disk.sda.out‚ 28.46484375 disk.drbd1817.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 0.46484375 disk.drbd2111.in‚0 disk.drbd1931.out‚0 disk.dm-0.out‚ 30.1103515625 disk.drbd1550.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 network.sent‚ 2.351447e+06 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1289.in‚0 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.rbd0.out‚0 disk.drbd1584.in‚0) disk./.usagePercent‚31.89570046937056" disk.nvme0n1.out‚ 30.1103515625 disk.drbd2078.in‚0 disk.sda1.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0$ disk./.usageGB‚570.3275184631348 disk.drbd2041.in‚0$ proc.memory.rssMB‚11253.33984375 cpu‚14.002015018024032 disk.sda2.out‚ 28.46484375 disk.sda.in‚0$ memory_percent‚41.21239185333252( proc.memory.availableMB‚ 38527.046875* proc.memory.percent‚17.171233892440796 proc.cpu.threads‚29 disk.drbd1938.in‚0 disk.drbd1673.out‚0 network.recv‚201079 disk.rbd0.in‚0 disk.sda2.in‚0Ž,€,›÷‚(: ƒŸÄÓ–½¹ gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚99.454' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚33.15133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|þ—Dø‚(:¹  ƒŸÄÓ•þì¹ disk.rbd0.in‚0 disk.drbd1226.in‚0$ proc.memory.rssMB‚11253.33984375 disk.drbd1584.in‚0 disk.sda.in‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1510.out‚0 disk.drbd1356.in‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.dm-0.in‚0 disk.drbd1510.in‚0 disk.nvme0n1.in‚0 disk.drbd1285.out‚0 proc.cpu.threads‚29 disk.drbd1289.in‚0 disk.drbd2118.in‚0* proc.memory.percent‚17.171233892440796 disk.drbd2041.out‚0" disk.nvme0n1.out‚ 30.2978515625 disk.rbd0.out‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.rbd1.out‚ 0.46484375 disk.sda2.in‚0 disk.drbd1817.in‚0 network.recv‚203338 disk.drbd1828.out‚0 disk.dm-0.out‚ 30.2978515625 disk.drbd1673.in‚0 disk.drbd1356.out‚0 disk.sda2.out‚ 28.69140625 disk.drbd1828.in‚0 disk.drbd1673.out‚0* disk./.usagePercent‚31.895702602748965( proc.memory.availableMB‚ 38526.796875 disk.drbd2111.out‚0 disk.drbd2118.out‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1550.out‚0 disk.drbd1938.out‚0 disk.drbd1285.in‚0$ disk./.usageGB‚570.3275566101074 disk.drbd1552.in‚0 disk.sda.out‚ 28.69140625 disk.sda1.out‚0 disk.drbd1817.out‚0$ memory_percent‚41.21277332305908 cpu‚13.996348402575293 network.sent‚ 2.396387e+06•“Ò$2ù- _wandbruntime‚1113  _runtime‚1113‹3ÊjÆúÀ $ _timestamp‚1.785794439473221e+09 _runtime‚1113.550184721 _step‚196 & lr-AdamW/pg1‚6.440469402677994e-05 & lr-AdamW/pg2‚6.440469402677994e-05 trainer/global_step‚2499U’Á ËûÅ _step‚196 & lr-AdamW/pg1‚6.440469402677994e-05 & lr-AdamW/pg2‚6.440469402677994e-05 trainer/global_step‚2499 $ _timestamp‚1.785794439473221e+09 _runtime‚1113.550184721Ä yû2ü- _wandbruntime‚1113  _runtime‚1113×3tÊý³O PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16612077 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17011936 @8DeviceStatsMonitor.on_trainzr/vï&_batch_start/num_device_alloc‚251 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2199490552832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9844897 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42921450200064 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42921433160704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41854975477472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16612079 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚2499 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚45282802419712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41854992516832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26396749 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.785794439474317e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45120836283904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7166774 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43008826 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16612077 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2199403123200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26406405 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2194020051820 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45282962191872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2259320953344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2194105854944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16612079 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43018484 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7167039 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45120940752896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42921433160704 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43023641238528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43023616334848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 _step‚197 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17011669 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42921450200064 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9844895 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0å°¾'þ¸O JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 _step‚197 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17011669 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42921450200064 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9844895 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16612077 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17011936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2199490552832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9844897 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42921450200064 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42921433160704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41854975477472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16612079 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚2499 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚45282802419712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41854992516832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26396749 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.785794439474317e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45120836283904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7166774 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43008826 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16612077 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2199403123200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26406405 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2194020051820 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45282962191872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2259320953344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2194105854944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16612079 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43018484 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7167039 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45120940752896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42921433160704 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43023641238528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43023616334848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130Å}}à±2ÿ- _wandbruntime‚1113  _runtime‚1113X‚·ÔÄ&€¾M ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45138987142656 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16618720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43035664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2200282742784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26416942 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7857944398427966e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9848832 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43035664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43025992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41871732689152 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41871715649792 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26416942 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17018515 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2260221978624 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45138987142656 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42938616963072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2260085020160 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2194897525042 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26407272 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 trainer/global_step‚2499 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44066613174834 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2194983328278 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16618720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16618722 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7169683 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43040848355328 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42938599923712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7169942 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16618722 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42938616963072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26407272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2200370179584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44066716017430 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251ßsíò: ÃM ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45138987142656 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16618720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43035664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2200282742784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26416942 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7857944398427966e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9848832 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43035664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43025992 E?DeviceStatsMonito«Ôr.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41871732689152 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41871715649792 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26416942 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17018515 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2260221978624 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45138987142656 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42938616963072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2260085020160 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2194897525042 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26407272 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 trainer/global_step‚2499 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44066613174834 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2194983328278 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16618720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16618722 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7169683 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43040848355328 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42938599923712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7169942 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16618722 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42938616963072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26407272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2200370179584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44066716017430 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251Æ»vš42‚- _wandbruntime‚1113  _runtime‚1113H¸˜oƒû /loss/FRt2/assignment_loss‚4.729814637452364 1loss/FBt1/assignment_loss‚0.36268773062183757 0loss/FBt2/assignment_loss‚0.2819301104924258 .loss/FRt2/detection_loss‚4.117845010943711 1loss/SRqqt1/detection_loss‚1.8168993950960035 /loss/FBt2/detection_loss‚4.5565427924023085 epoch‚0 2loss/SRqqt1/assignment_loss‚1.4087040511365154 .loss/FRt1/detection_loss‚2.765711988721575 1loss/SRqqt2/detection_loss‚2.9499661092886678 &loss/total_loss‚36.964221904500036 % _timestamp‚1.7857944398445463e+09 _runtime‚ 1113.92132466 /loss/FRt1/assignment_loss‚3.317214324360802 2loss/SRqqt2/assignment_loss‚1.9385599450009767 _step‚199 .loss/FBt1/detection_loss‚8.718345808982852 trainer/global_step‚2499ÍWò„€ 0loss/FBt2/assignment_loss‚0.2819301104924258 .loss/FRt2/detection_loss‚4.117845010943711 1loss/SRqqt1/detection_loss‚1.8168993950960035 /loss/FBt2/detection_loss‚4.5565427924023085 epoch‚0 2loss/SRqqt1/assignment_loss‚1.4087040511365154 .loss/FRt1/detection_loss‚2.765711988721575 1loss/SRqqt2/detection_loss‚2.9499661092886678 &loss/total_loss‚36.964221904500036 % _timestamp‚1.7857944398445463e+09 _runtime‚ 1113.92132466 /loss/FRt1/assignment_loss‚3.317214324360802 2loss/SRqqt2/assignment_loss‚1.9385599450009767 _step‚199 .loss/FBt1/detection_loss‚8.718345808982852 trainer/global_step‚2499 /loss/FRt2/assignment_loss‚4.729814637452364 1loss/FBt1/assignment_loss‚0.36268773062183757Ç{ò]1›…‚(: ’ŸÄÓŒÄ߸ gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚94.367' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚31.45566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gä°VC†‚(:¸  ’ŸÄÓø) disk./.usagePercent‚31.89570046937056 disk.drbd2041.out‚0 disk.drbd1965.out‚0 proc.cpu.threads‚29 disk.drbd1285.out‚0 network.sent‚ 2.399391e+06 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚ 29.05078125 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd1550.out‚0 disk.drbd1356.in‚0 disk.rbd0.out‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd2118.out‚0 disk.dm-0.in‚0 disk.drbd1938.out‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0" disk.nvme0n1.out‚ 30.7158203125 disk.drbd1965.in‚0 disk.rbd1.out‚ 0.46484375 disk.drbd1828.in‚0 disk.drbd1817.out‚0$ disk./.usageGB‚570.3275184631348 disk.sda.out‚ 29.05078125 disk.sda.in‚0 disk.drbd1510.out‚0 disk.drbd1289.in‚0( proc.memory.availableMB‚ 38526.734375 disk.sda1.in‚0 disk.dm-0.out‚ 30.7158203125 disk.drbd1550.in‚0 disk.drbd1226.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1584.in‚0* proc.memory.percent‚17.171233892440796 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.rbd1.in‚0 disk.drbd2111.out‚0$ proc.memory.rssMB‚11253.33984375 disk.drbd2041.in‚0 disk.drbd1931.in‚0 disk.drbd1817.in‚0 disk.drbd1510.in‚0 disk.nvme0n1.in‚0 disk.drbd1285.in‚0 disk.drbd1931.out‚0 disk.drbd1673.in‚0 cpu‚13.990745028396722 disk.sda1.out‚0 disk.drbd1828.out‚0$ memory_percent‚41.21286869049072 network.recv‚2044048´ƒP2‡- _wandbruntime‚1135  _runtime‚1135¤«ÇQƈÀ & lr-AdamW/pg1‚6.569330335104525e-05 & lr-AdamW/pg2‚6.569330335104525e-05 trainer/global_step‚2549 $ _timestamp‚1.785794461786121e+09 _runtime‚1135.864649864 _step‚200ZHSˉŠ_step‚200 & lr-AdamW/pg1‚6.569330335104525e-05 & lr-AdamW/pg2‚6.569330335104525e-05 trainer/global_step‚2549 $ _timestamp‚1.785794461786121e+09 _runtime‚1135.864649864Ⱥ@Ï.2Š- _wandbruntime‚1135  _runtime‚1135LO4¼'‹¶O ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2243399258624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43868978 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 % _timestamp‚1.7857944617868872e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2237994318452 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2243486688256 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46188347240448 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43883997078528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚43779788350464 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16944229 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17351534 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2304507836928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚42691984093472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10041745 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _runtime‚1135.864749047 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16944227 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2304375065600 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44929995451284 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44929892608800 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46023170569728 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚43779771311104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10041747 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚43779771311104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46023170569728 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16944227 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26924751 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46023275038720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2243399258624 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43878636 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16944229 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46023275038720 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚42692001132832_cÁ'Œ»O E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚43779788350464 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16944229 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17351534 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2304507836928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚42691984093472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10041745 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _runtime‚1135.864749047 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16944227 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2304375065600 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44929995451284 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44929892608800 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46023170569728 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚43779771311104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10041747 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚43779771311104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46023170569728 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16944227 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26924751 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46023275038720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2243399258624 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43878636 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16944229 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46023275038720 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚42692001132832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2243399258624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43868978 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 % _timestamp‚1.7857944617868872e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2237994318452 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2243486688256 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46188347240448 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43883997078528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0É12- _wandbruntime‚1136  _runtime‚1136M[ƒ€TŽ¿M SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2305280308736 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16950870 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_p&{ùq#ool.freed‚ 2238785988550 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2244366315008 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43796938074112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10045684 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7312309 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26935274 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 _step‚202 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚46041216952320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43901179291648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7857944621567152e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44947510254342 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2238871791786 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2549 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2244366315008 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16950872 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚46041321428480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44947613096938 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚42708724265792 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43886144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43886144 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46206613073920 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43895816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7312552 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46206459600384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43796938074112 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚42708741305152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 1136.23405587 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10045682 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17357991 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16950870 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26935274 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26944944 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43895816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160_åÔÊ&ÄM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44947510254342 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2238871791786 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2549 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2244366315008 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16950872 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚46041321428480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44947613096938 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚42708724265792 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43886144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43886144 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46206613073920 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43895816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7312552 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46206459600384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43796938074112 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚42708741305152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 1136.23405587 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10045682 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17357991 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16950870 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26935274 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26944944 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43895816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2305280308736 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16950870 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2238785988550 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2244366315008 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43796938074112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10045684 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7312309 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26935274 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 _step‚202 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚46041216952320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43901179291648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7857944621567152e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0Êεöä2- _wandbruntime‚1136  _runtime‚1136Þý ‘ý 2loss/SRqqt2/assignment_loss‚1.8709321747655454 1loss/SRqqt2/detection_loss‚2.7449702365274016 epoch‚0 _runtime‚ 1136.23521008 _step‚203 0loss/FRt1/assignment_loss‚3.5669369195021834 /loss/FRt2/assignment_loss‚4.621912869540128 0loss/FBt1/assignment_loss‚0.2648200769714701 /loss/FRt1/detection_loss‚2.8026793654642854 % _timestamp‚1.7857944621584487e+09 1loss/FBt2/assignment_loss‚0.06136795619650828 .loss/FRt2/detection_loss‚3.853566259597287 /loss/FBt2/detection_loss‚3.8396834283890353 trainer/global_step‚2549 2loss/SRqqt1/assignment_loss‚1.4527481638414914 1loss/SRqqt1/detection_loss‚1.6711496258754186 /loss/FBt1/detection_loss‚15.872392733891806 %loss/total_loss‚42.62315981056256ƒˆS‡’‚ 1loss/FBt2/assignment_loss‚0.06136795619650828 .loss/FRt2/detection_loss‚3.853566259597287 /loss/FBt2/detection_loss‚3.8396834283890353 trainer/global_step‚2549 2loss/SRqqt1/assignment_loss‚1.4527481638414914 1loss/SRqqt1/detection_loss‚1.6711496258754186 /loss/FBt1/detection_loss‚15.872392733891806 %loss/total_loss‚42.62315981056256 2loss/SRqqt2/assignment_loss‚1.8709321747655454 1loss/SRqqt2/detection_loss‚2.7449702365274016 epoch‚0 _runtime‚ 1136.23521008 _step‚203 0loss/FRt1/assignment_loss‚3.5669369195021834 /loss/FRt2/assignment_loss‚4.621912869540128 0loss/FBt1/assignment_loss‚0.2648200769714701 /loss/FRt1/detection_loss‚2.8026793654642854 % _timestamp‚1.7857944621584487e+09ËLÞiœ“‚(:‘ ¡ŸÄÓ²‰÷¸ gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚127.364' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚42.45466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06£žÁ?”‚(:´  ¡ŸÄӕо¹ disk.drbd2118.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1965.out‚0 disk.drbd1673.out‚0 disk.rbd1.out‚ 0.46484375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0( proc.memory.availableMB‚ 38526.140625 disk.drbd2041.in‚0 disk.drbd1673.in‚0 disk.drbd1817.out‚0 disk.drbd1584.in‚0* proc.memory.percent‚17.171233892440796 network.recv‚206750 disk.drbd1552.in‚0( disk./.usagePercent‚31.8957403635468 disk.drbd2041.out‚0 disk.drbd1226.in‚0 network.sent‚ 2.4443e+06 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.sda.out‚ 29.41015625 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0$ proc.memory.rssMB‚11253.33984375 disk.sda2.out‚ 29.41015625 disk.drbd2111.out‚0" disk.nvme0n1.out‚ 30.8486328125$ disk./.usageGB‚570.3282318115234 disk.rbd0.in‚0 disk.sda1.in‚0 cpu‚13.98515700465943 disk.drbd1285.in‚0 disk.rbd1.in‚0 disk.drbd1828.out‚0 disk.drbd1931.out‚0 disk.dm-0.in‚0 disk.drbd1285.out‚0 disk.drbd1938.in‚0 disk.drbd1817.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 proc.cpu.threads‚29 disk.nvme0n1.in‚0$ memory_percent‚41.21377468109131 disk.drbd1938.out‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.dm-0.out‚ 30.8486328125 disk.drbd1552.out‚0 disk.drbd1965.in‚0Ñ]™È•‚(:’ °ŸÄÓá掹 gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚118.614' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.538000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²t_èD–‚(:¹  °ŸÄÓÁùع disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1356.out‚0 disk.drbd1965.in‚0 disk.drbd1226.out‚0" disk.nvme0n1.out‚ 31.1181640625 disk.sda.in‚0 disk.drbd2041.in‚0 disk.drbd1931.out‚0$ proc.memory.rssMB‚11253.33984375 disk.drbd2078.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 network.sent‚ 2.489261e+06 disk.drbd1285.out‚0 disk.sda2.out‚ 29.69921875 disk.sda.out‚ 29.69921875 proc.cpu.threads‚29 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.drbd1828.out‚0 disk.drbd1584.in‚0$ disk./.usageGB‚570.3281898498535 disk.drbd1510.out‚0 disk.drbd1356.in‚0 disk.drbd1550.out‚0 disk.drbd1673.in‚0 disk.drbd1552.in‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0* disk./.usagePercent‚31.895738016830542 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd2041.out‚0 disk.drbd1817.in‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.rbd0.in‚0 disk.drbd2111.in‚0( proc.memory.availableMB‚ 38526.140625* proc.memory.percent‚17.171233892440796 cpu‚13.980014017108122 disk.sda1.out‚0$ memory_percent‚41.21377468109131 disk.rbd1.out‚ 0.46484375 disk.dm-0.in‚0 disk.dm-0.out‚ 31.1181640625 disk.drbd1510.in‚0 network.recv‚209158 disk.drbd1938.in‚0 disk.drbd1817.out‚0 disk.nvme0n1.in‚0 disk.drbd1289.in‚0 disk.drbd2118.out‚0°» 2—- _wandbruntime‚1158  _runtime‚1158›á~ǘÁ & lr-AdamW/pg1‚6.698191267531055e-05 & lr-AdamW/pg2‚6.698191267531055e-05 trainer/global_step‚2599 % _timestamp‚1.7857944841349711e+09 _runtime‚1158.211936857 _step‚204³ÂÌ™Æ % _timestamp‚1.7857944841349711e+09 _runtime‚1158.211936857 _step‚204 & lr-AdamW/pg1‚6.698191267531055e-05 & lr-AdamW/pg2‚6.698191267531055e-05 trainer/global_step‚2599Ì—D÷Þ2š- _wandbruntime‚1158  _runtime‚1158§ñés›¶O NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44638126500864 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚44744328014848 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚45810789688308 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 trainer/global_step‚2599 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17276379 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27462409 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43529009748832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2287482823680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17276379 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46925504855552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43528992709472 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46925504855552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7452629 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2349697612288 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27452753 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46925609324544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17690983 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46925504855552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43528992709472 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46925504855552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7452629 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2349697612288 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27452753 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46925609324544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17690983 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2288362450432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44755968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43545732881792 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27472946 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2288275013632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44755968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚45828510176446 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2599 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27463276 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17697754 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7857944845160549e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44655293263872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17283020 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17283022 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27472946 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44655276224512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17283020 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚45828407333850 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7455456 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚47112158763008 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2282674452058 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44655276224512 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17283022 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44746296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10242532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚44761535131648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43545749921152 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚47112007386624—ʸÊ&ŸÄM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _step‚206 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27463276 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚46943551238144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2288362450432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44755968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43545732881792 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27472946 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2288275013632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44755968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚45828510176446 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2599 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27463276 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17697754 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7857944845160549e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44655293263872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17283020 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17283022 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27472946 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44655276224512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17283020 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚45828407333850 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7455456 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚47112158763008 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2282674452058 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44655276224512 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17283022 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44746296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10242532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚44761535131648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43545749921152 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚47112007386624 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2288362450432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032ÎDÝh€  - _wð2 5'andbruntime‚1158  _runtime‚1158¶BL¼¡û _runtime‚1158.594735311 2loss/SRqqt2/assignment_loss‚1.7436709724632757 1loss/FBt2/assignment_loss‚0.06242190828065346 1loss/SRqqt1/detection_loss‚1.5577299679039995 -loss/FBt1/detection_loss‚7.11774239756845 2loss/SRqqt1/assignment_loss‚1.4747255489283757 0loss/FBt1/assignment_loss‚0.5836510920304466 1loss/SRqqt2/detection_loss‚2.9482631488277056 %loss/total_loss‚34.79102301031973 /loss/FRt2/assignment_loss‚4.640894602422845 /loss/FRt2/detection_loss‚3.5481558932016974 .loss/FBt2/detection_loss‚4.066443632301013 trainer/global_step‚2599 $ _timestamp‚1.785794484517919e+09 _step‚207 0loss/FRt1/assignment_loss‚3.4686199101534756 /loss/FRt1/detection_loss‚3.5787039362377935 epoch‚0â§Û¢€ 1loss/FBt2/assignment_loss‚0.06242190828065346 1loss/SRqqt1/detection_loss‚1.5577299679039995 -loss/FBt1/detection_loss‚7.11774239756845 2loss/SRqqt1/assignment_loss‚1.4747255489283757 0loss/FBt1/assignment_loss‚0.5836510920304466 1loss/SRqqt2/detection_loss‚2.9482631488277056 %loss/total_loss‚34.79102301031973 /loss/FRt2/assignment_loss‚4.640894602422845 /loss/FRt2/detection_loss‚3.5481558932016974 .loss/FBt2/detection_loss‚4.066443632301013 trainer/global_step‚2599 $ _timestamp‚1.785794484517919e+09 _step‚207 0loss/FRt1/assignment_loss‚3.4686199101534756 /loss/FRt1/detection_loss‚3.5787039362377935 epoch‚0 _runtime‚1158.594735311 2loss/SRqqt2/assignment_loss‚1.7436709724632757ÏR•ÐP‘£‚(:† ¿ŸÄÓú‚Š¹ gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚129.108' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚43.036 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M&/LC¤‚(:¸  ¿ŸÄÓ³¡Ò¹ disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1510.out‚0 proc.cpu.threads‚29 disk.rbd1.in‚0 disk.sda2.out‚ 29.9453125 disk.drbd1817.in‚0 disk.drbd1550.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1938.in‚0 disk.nvme0n1.in‚0$ disk./.usageGB‚570.3282127380371 disk.drbd1931.out‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.sda.out‚ 29.9453125 disk.dm-0.out‚ 31.3447265625 network.sent‚ 2.492269e+06 disk.drbd2041.in‚0 disk.dm-0.in‚0 disk.drbd1673.out‚0$ memory_percent‚41.21312499046326* proc.memory.availableMB‚38526.56640625 cpu‚13.97479975276626 disk.rbd1.out‚ 0.46484375 disk.sda.in‚0 disk.drbd1552.out‚0$ proc.memory.rssMB‚11253.33984375 network.recv‚210109 disk.drbd1226.in‚0* disk./.usagePercent‚31.895739296857588 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.sda2.in‚0 disk.drbd1828.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd1965.out‚0 disk.drbd1226.out‚0 disk.drbd1817.out‚0 disk.drbd1510.in‚0" disk.nvme0n1.out‚ 31.3447265625 disk.drbd1285.in‚0 disk.drbd1356.out‚0* proc.memory.percent‚17.171233892440796*$´™2¥- _wandbruntime‚1180  _runtime‚1180qA|´Æ¦À & lr-AdamW/pg1‚6.827052199957586e-05 & lr-AdamW/pg2‚6.827052199957586e-05 trainer/global_step‚2649 $ _timestamp‚1.785794505942357e+09 _runtime‚1180.019317307 _step‚208䔼˧Š& lr-AdamW/pg1‚6.827052199957586e-05 & lr-AdamW/pg2‚6.827052199957586e-05 trainer/global_step‚2649 $ _timestamp‚1.785794505942357e+09 _runtime‚1180.019317307 _step‚208ШÐøç2¨- _wandbruntime‚1180  _runtime‚1180MŒk9»'©µO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 trainer/global_step‚2649 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2331478959104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 _step‚209 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45496447611904 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚46691686767816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚44366018364832 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17608527 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚47827839141376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45496464651264 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚47827943610368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2331391529472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7594934 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45496447611904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17608529 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27990411 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2331391529472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45589282 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17608529 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚47827943610368 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45496464651264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚46691789610300 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚1180.020671659 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚44366001325472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27990411 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7595176 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45604708758528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚47827839141376 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚47999599490560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10435447 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27980755 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2331478959104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17608527 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2325685442344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45598940 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18030623 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 trainer/global_step‚2649 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2331478959104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 _step‚209 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45496447611904 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚46691686767816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚44366018364832 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17608527 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚47827839141376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45496464651264 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚47827943610368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2331391529472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7594934 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45496447611904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17608529 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27990411 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2331391529472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45589282 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17608529 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚47827943610368 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45496464651264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚46691789610300 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚1180.020671659 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚44366001325472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27990411 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7595176 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45604708758528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚47827839141376 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚47999599490560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10435447 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27980755 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2331478959104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17608527 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2325685442344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45598940 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130Ñܧa­2«- _wandbruntime‚1180  _runtime‚1180ô®RQ ¬ÀM VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚46709407255954 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45606448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚45513614374912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚44382758537152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2332358585856 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45621915875328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚48017552922112 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2395803103232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17615172 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _runtime‚1180.394853681 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚47845990000128 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48017718978560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45616120 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17615170 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7597943 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2326562915566 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2395661950464 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚44382741497792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45513631414272 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17615170 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 % _timestamp‚1.7857945063174999e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141152768 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7597701 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27991278 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18037327 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚47845885523968 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10439384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2332358585856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45621890971648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2326648718802 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45606448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 _step‚210 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17615172 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45616120 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚45513631414272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28000948 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 R?DeviceStatsMonitor.on_train_batch_ehô£ßÅnd/requested_bytes.all.freed‚46709304413358 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚45513614374912 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚2649 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18037083 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1Šk(Ë&­ÅM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17615172 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _runtime‚1180.394853681 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚47845990000128 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48017718978560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45616120 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17615170 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7597943 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2326562915566 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2395661950464 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚44382741497792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45513631414272 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17615170 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 % _timestamp‚1.7857945063174999e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141152768 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7597701 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27991278 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18037327 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚47845885523968 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10439384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2332358585856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45621890971648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2326648718802 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45606448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 _step‚210 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17615172 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45616120 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚45513631414272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28000948 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚46709304413358 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚45513614374912 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚2649 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18037083 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚46709407255954 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45606448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚45513614374912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚44382758537152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2332358585856 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45621915875328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚48017552922112 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2395803103232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18Ò@>Êr2®- _wandbruntime‚1180  _runtime‚1180O좯þ /loss/FBt2/detection_loss‚3.3290452524611056 %loss/total_loss‚33.88958752418715 _runtime‚1180.396099852 _step‚211 2loss/SRqqt1/assignment_loss‚1.4646667058311276 epoch‚0 0loss/FRt1/assignment_loss‚3.4512326762646057 2loss/SRqqt2/assignment_loss‚1.9195999205112457 1loss/FBt2/assignment_loss‚0.03177766489688744 1loss/SRqqt1/detection_loss‚1.8261373907061618 .loss/FBt1/detection_loss‚7.643620583746168 trainer/global_step‚2649 % _timestamp‚1.7857945063193135e+09 /loss/FRt2/assignment_loss‚4.583761701217065 0loss/FBt1/assignment_loss‚0.7153976261615753 /loss/FRt1/detection_loss‚3.4035252494902553 1loss/SRqqt2/detection_loss‚2.2041591625217034 /loss/FRt2/detection_loss‚3.3166635903792505ýÅ3 °ƒ /loss/FRt2/assignment_loss‚4.583761701217065 0loss/FBt1/assignment_loss‚0.7153976261615753 /loss/FRt1/detection_loss‚3.4035252494902553 1loss/SRqqt2/detection_loss‚2.2041591625217034 /loss/FRt2/detection_loss‚3.3166635903792505 /loss/FBt2/detection_loss‚3.3290452524611056 %loss/total_loss‚33.88958752418715 _runtime‚1180.396099852 _step‚211 2loss/SRqqt1/assignment_loss‚1.4646667058311276 epoch‚0 0loss/FRt1/assignment_loss‚3.4512326762646057 2loss/SRqqt2/assignment_loss‚1.9195999205112457 1loss/FBt2/assignment_loss‚0.03177766489688744 1loss/SRqqt1/detection_loss‚1.8261373907061618 .loss/FBt1/detection_loss‚7.643620583746168 trainer/global_step‚2649 % _timestamp‚1.7857945063193135e+09Ó Ð䜱‚(:‘ ΟÄÓ…öḠgpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚117.001' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.00033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š¨œÀG²‚(:¼  ΟÄÓ·ƒ§¹ network.sent‚ 2.537315e+06 disk.drbd2078.in‚0 disk.drbd1550.in‚0 disk.drbd2111.in‚0 disk.drbd1584.out‚0 disk.sda.in‚0$ memory_percent‚41.21374487876892 disk.rbd1.in‚0 network.recv‚212540$ disk./.usageGB‚570.3281936645508 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd2118.in‚0 disk.drbd2078.out‚0 disk.drbd1817.out‚0* disk./.usagePercent‚31.895738230168387 disk.sda2.in‚0 disk.drbd2118.out‚0* proc.memory.availableMB‚38526.16015625 disk.drbd1965.out‚0 disk.dm-0.in‚0 disk.drbd1938.in‚0 disk.drbd1226.in‚0 disk.nvme0n1.in‚0 disk.drbd1285.out‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 0.55078125 disk.drbd1817.in‚0 disk.drbd1584.in‚0 disk.drbd1226.out‚0 disk.sda2.out‚ 30.34765625 disk.sda1.out‚0 cpu‚13.96961715085686 disk.drbd1285.in‚0 disk.drbd1356.in‚0 disk.drbd1965.in‚0 disk.dm-0.out‚31.59130859375 disk.drbd1289.in‚0* proc.memory.percent‚17.171233892440796 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd2041.in‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.rbd0.in‚0 disk.drbd1828.in‚0 disk.drbd1931.out‚0 disk.sda.out‚ 30.34765625 disk.drbd1938.out‚0# disk.nvme0n1.out‚31.59130859375 disk.drbd1356.out‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0 proc.cpu.threads‚29 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1550.out‚0 disk.drbd2041.out‚0$ proc.memory.rssMB‚11253.33984375EIÕ°Ž³‚(:ƒ ÝŸÄÓâ̇¹ gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚98.28' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚32.76 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZÔ+F´‚(:»  ÝŸÄÓ¾ˆÔ¹$ disk./.usageGB‚570.3283348083496# disk.nvme0n1.out‚31.78271484375 disk.drbd2111.out‚0 disk.drbd2118.out‚0 cpu‚13.9644428038558 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.sda2.out‚ 30.69140625 disk.drbd2111.in‚0 proc.cpu.threads‚29 disk.drbd1510.out‚0 disk.nvme0n1.in‚0 disk.drbd1828.in‚0$ proc.memory.rssMB‚11253.33984375 disk.rbd1.out‚ 0.61328125* proc.memory.availableMB‚38526.16015625 disk.drbd2041.in‚0$ memory_percent‚41.21374487876892 disk.drbd1289.in‚0 disk.drbd1550.in‚0 disk.drbd1828.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd2041.out‚0* proc.memory.percent‚17.171233892440796 disk.drbd1965.in‚0 disk.drbd2078.in‚0 network.sent‚ 2.582244e+06 disk.drbd1510.in‚0 disk.drbd1285.in‚0 disk.drbd2078.out‚0 disk.drbd1550.out‚0 disk.drbd1226.out‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 network.recv‚214800* disk./.usagePercent‚31.895746123668502 disk.drbd1817.out‚0 disk.drbd1285.out‚0 disk.drbd1356.in‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚0 disk.drbd1584.out‚0 disk.sda.in‚0 disk.sda.out‚ 30.69140625 disk.dm-0.in‚0 disk.dm-0.out‚31.78271484375 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.sda1.out‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1673.in‚0Þó2µ- _wandbruntime‚1202  _runtime‚1202×s()Ŷ¿ _runtime‚ 1202.32982027 _step‚212 & lr-AdamW/pg1‚6.955913132384115e-05 & lr-AdamW/pg2‚6.955913132384115e-05 trainer/global_step‚2699 $ _timestamp‚1.785794528252864e+09†“EzÊ·Ä trainer/global_step‚2699 $ _timestamp‚1.785794528252864e+09 _runtime‚ 1202.32982027 _step‚212 & lr-AdamW/pg1‚6.955913132384115e-05 & lr-AdamW/pg2‚6.955913132384115e-05Ô‹’2¸- _wandbruntime‚1202  _runtime‚1202˜©¼'¹¶O TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17940679 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17940677 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28518413 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚46354785762304 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2439944290816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10632297 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48730277896192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28518413 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2375475094528 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚48730277896192 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48730173427200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47572686689808 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17940677 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17940679 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7738483 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7857945282538717e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 trainer/global_step‚2699 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48730173427200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚46354785762304 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚45203009941472 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45203026980832 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18370780 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚46354785762304 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2439944290816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10632297 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 YLDeviCâ00_ceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48730277896192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28518413 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2375475094528 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚48730277896192 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48730173427200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47572686689808 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17940677 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17940679 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7738483 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7857945282538717e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 trainer/global_step‚2699 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48730173427200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚46354785762304 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚45203009941472 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45203026980832 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18370780 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18377220 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46466600 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10636234 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10636232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚46371952525312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46476272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7857945286239383e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48748324285952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48923250769408 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47590304335462 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47590201492866 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46482271715328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48748219809792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28528950 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7741273 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚45219767153152 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46466600 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2440979054080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚1202.701259895 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18377507 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168153600 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚46371952525312 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46476272 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17947322 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚46482246811648 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856«èË&½ÅM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46476272 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17947322 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚46482246811648 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 trainer/global_step‚2699 _step‚214 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2376267284480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17947320 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚48748324285952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28528950 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28519280 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚46371969564672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17947322 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18377220 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46466600 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10636234 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10636232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚46371952525312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46476272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7857945286239383e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48748324285952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48923250769408 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47590304335462 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47590201492866 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46482271715328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48748219809792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28528950 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7741273 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚45219767153152 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46466600 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2440979054080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚1202.701259895 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18377507 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168153600 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚46371952525312 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47319244 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚47325395534848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2419471229952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚47325420438528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2419471229952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7880891 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2485083414528 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18272827 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2485214088704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚46040018557472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2413548172484 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 trainer/global_step‚2749 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚49632507713024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18272829 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47213140952064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2413462369360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29036759 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49632507713024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7880629 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚48453480926832 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29046415 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29036759 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚49810634527232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29046415 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚47213140952064 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18709774 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 $ _timestamp‚1.785794550389741e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18272827 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18272829 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18710038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚48453583769316 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2419383800320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10829145 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚49810478949376 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _step‚217 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚49632612182016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚46040035596832 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47309586% ¿'ȹO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚49632507713024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18272829 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47213140952064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2413462369360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29036759 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49632507713024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7880629 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚48453480926832 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29046415 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29036759 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚49810634527232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29046415 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚47213140952064 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18709774 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 $ _timestamp‚1.785794550389741e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18272827 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18272829 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18710038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚48453583769316 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2419383800320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10829145 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚49810478949376 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _step‚217 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚49632612182016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚46040035596832 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47309586 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 _runtime‚ 1224.47060561 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47319244 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚47325395534848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2419471229952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚47325420438528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2419471229952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7880891 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2485083414528 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18272827 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2485214088704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚46040018557472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2413548172484 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 trainer/global_step‚2749 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47336424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29056952 _step‚218 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49650554095616 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚49650658571776 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7883809 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29056952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2486116056064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46056775769152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29047282 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚48471201414970 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7883563 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47336424 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47326752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10833084 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47342627555328 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2414425645818 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚49650658571776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚48471098572374 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18716645 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47342602651648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7857945507555113e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18279470 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚47230307715072 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2420263419904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚1224.832851257 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2414339842582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18716893 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47230290675712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18279472 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320Mâö—ÑËÅM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29056952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2486116056064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46056775769152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29047282 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚48471201414970 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7883563 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47336424 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47326752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10833084 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47342627555328 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2414425645818 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚49650658571776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚48471098572374 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18716645 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47342602651648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7857945507555113e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18279470 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚47230307715072 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2420263419904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚1224.832851257 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2414339842582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18716893 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47230290675712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18279472 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2420263419904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47336424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29056952 _step‚218 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49650554095616 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚49650658571776 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7883809 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712Ús2Ì- _wandbruntime‚1224  _runtime‚1224)8ðÿÍù /loss/FBt1/assignment_loss‚0.346066688222345 1loss/SRqqt1/detection_loss‚1.6638736526830575 .loss/FBt1/detection_loss‚7.166968774795532 $ _timestamp‚1.785794550757241e+09 _step‚219 2loss/SRqqt1/assignment_loss‚1.3970073483517913 /loss/FRt1/detection_loss‚2.8416349600921285 .loss/FRt2/detection_loss‚3.518288681472557 %loss/total_loss‚34.02647125172638 epoch‚0 _runtime‚1224.834010749 /loss/FRt1/assignment_loss‚3.684830891325119 2loss/SRqqt2/assignment_loss‚1.9450801847768682 1loss/FBt2/assignment_loss‚0.06518612008259975 /loss/FRt2/assignment_loss‚4.529145665790724 0loss/SRqqt2/detection_loss‚4.259408664255618 /loss/FBt2/detection_loss‚2.6089796198780353 trainer/global_step‚2749 ¥ÐÎþ 2loss/SRqqt1/assignment_loss‚1.3970073483517913 /loss/FRt1/detection_loss‚2.8416349600921285 .loss/FRt2/detection_loss‚3.518288681472557 %loss/total_loss‚34.02647125172638 epoch‚0 _runtime‚1224.834010749 /loss/FRt1/assignment_loss‚3.684830891325119 2loss/SRqqt2/assignment_loss‚1.9450801847768682 1loss/FBt2/assignment_loss‚0.06518612008259975 /loss/FRt2/assignment_loss‚4.529145665790724 0loss/SRqqt2/detection_loss‚4.259408664255618 /loss/FBt2/detection_loss‚2.6089796198780353 trainer/global_step‚2749 /loss/FBt1/assignment_loss‚0.346066688222345 1loss/SRqqt1/detection_loss‚1.6638736526830575 .loss/FBt1/detection_loss‚7.166968774795532 $ _timestamp‚1.785794550757241e+09 _step‚219Û\ÙévÏ‚(:’ ûŸÄÓË“™¹ gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚125.317' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚41.772333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èŒuáFЂ(:»  ûŸÄÓÏôÞ¹ disk.dm-0.in‚0 disk.nvme0n1.in‚0* proc.memory.availableMB‚38526.05078125 disk.sda.out‚ 31.296875 disk.dm-0.out‚32.45556640625 disk.drbd1289.out‚0 disk.drbd1550.out‚0# disk.nvme0n1.out‚32.45556640625 disk.drbd1552.in‚0 disk.drbd2111.in‚0 disk.drbd1931.out‚0 disk.rbd1.out‚ 0.6455078125 network.sent‚ 2.630208e+06 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1289.in‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0$ proc.memory.rssMB‚11253.33984375 disk.drbd1226.in‚0 disk.drbd1285.out‚0$ memory_percent‚41.21391177177429* disk./.usagePercent‚31.895745056979298 disk.rbd0.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 cpu‚13.954949008198623 disk.drbd1817.out‚0 disk.drbd1510.in‚0 disk.drbd1285.in‚0 disk.drbd1356.out‚0 disk.sda2.in‚0 disk.sda1.out‚0 proc.cpu.threads‚29 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 network.recv‚218130 disk.drbd1226.out‚0 disk.drbd1931.in‚0 disk.drbd1673.in‚0 disk.drbd1828.in‚0 disk.drbd1965.in‚0$ disk./.usageGB‚570.3283157348633 disk.drbd2041.out‚0 disk.drbd1938.in‚0 disk.drbd1817.in‚0 disk.rbd1.in‚0* proc.memory.percent‚17.171233892440796 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 31.296875 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.sda.in‚0.ìuãœÑ‚(:‘ Š ÄÓ¿Žö¸ gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚119.489' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.82966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nãµPTÒ‚(:É  Š ÄÓå½¹ cpu‚13.951753562361771 disk.drbd1817.in‚0 disk.drbd1931.out‚0$ proc.memory.rssMB‚11253.33984375 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1289.out‚0 proc.cpu.threads‚29$ disk./.usageGB‚570.3286437988281 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1285.in‚0 disk.rbd0.out‚0 disk.drbd1289.in‚0 disk.drbd2118.in‚0 disk.sda.in‚0* disk./.usagePercent‚31.895763404033605 disk.drbd1356.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1938.in‚0# disk.nvme0n1.out‚32.98681640625 disk.drbd1552.in‚0 disk.sda2.in‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0& proc.memory.availableMB‚ 38525.8125 disk.drbd1550.in‚0 disk.drbd2111.out‚0 disk.drbd1938.out‚0 network.sent‚ 2.675366e+06 disk.rbd0.in‚0 disk.sda2.out‚ 32.63671875 disk.drbd2041.in‚0 disk.drbd2041.out‚0 network.recv‚220489 disk.nvme0n1.in‚0.734375 disk.drbd1965.in‚0 disk.rbd1.out‚ 0.6455078125 disk.sda1.out‚0 disk.drbd1673.in‚0$ memory_percent‚41.21427536010742 disk.drbd1226.out‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.sda.out‚ 32.63671875 disk.sda1.in‚0 disk.drbd2111.in‚0 disk.dm-0.in‚0.734375 disk.drbd1673.out‚0 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.dm-0.out‚32.98681640625 disk.drbd1931.in‚0* proc.memory.percent‚17.171233892440796 disk.drbd1817.out‚0 disk.drbd1584.in‚0 º›¼2Ó- _wandbruntime‚1246  _runtime‚1246°4· ÇÔÁ & lr-AdamW/pg1‚7.213634997237176e-05 & lr-AdamW/pg2‚7.213634997237176e-05 trainer/global_step‚2799 % _timestamp‚1.7857945725364301e+09 _runtime‚1246.613420529 _step‚220# vÌÕÆ _step‚220 & lr-AdamW/pg1‚7.213634997237176e-05 & lr-AdamW/pg2‚7.213634997237176e-05 trainer/global_step‚2799 % _timestamp‚1.7857945725364301e+09 _runtime‚1246.613420529Ü—#0c2Ö- _wandbruntime‚1246  _runtime‚1246iÍT½¼'׶O SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8023817 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2463467365376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48179396 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 _step‚221 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚50534841998848 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚48071462063104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29574417 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2457436635992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚48071462063104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2457350832868 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50716135578112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚49334480848824 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7857945725375566e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2463379935744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18604977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2463467365376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19049812 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚48185776278528 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚46877027173472 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚50534946467840 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2530224431104 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48071479102464 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚50534946467840 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18604977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8024055 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50715975805952 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11025995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18604979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19050052 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124S)GÁ'Ø»O A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2463379935744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚48071479102464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11025997 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 _runtime‚1246.614858487 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29564761 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29564761 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚46877044212832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48185751374848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18604979 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 _step‚221 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚50534841998848 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚48071462063104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29574417 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2457436635992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚48071462063104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2457350832868 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50716135578112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚49334480848824 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7857945725375566e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2463379935744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18604977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2463467365376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19049812 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚48185776278528 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚46877027173472 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚50534946467840 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2530224431104 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48071479102464 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚50534946467840 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18604977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8024055 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50715975805952 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11025995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18604979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19050052 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8023817 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2463467365376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48179396 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8026885 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46893784385152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2464346992128 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48186904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18611620 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚50552888381440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29584954 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50734248569856 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2464346992128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19056569 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18611620 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2531130313216 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29575284 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2464259555328 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8026637 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚50552992857600 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2464259555328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19056819 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11029934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48196576 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48196576 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48186904 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚48088628826112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚50552992857600 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚48088645865472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2458314109326 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11029932 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46893767345792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚2799 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29575284 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚48088645865472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887‰Š=ÙË&ÛÅM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18611622 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29584954 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚48088628826112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚48202958491648 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8026885 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46893784385152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2464346992128 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48186904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18611620 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚50552888381440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29584954 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50734248569856 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2464346992128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19056569 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18611620 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2531130313216 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29575284 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2464259555328 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8026637 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚50552992857600 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2464259555328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19056819 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11029934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48196576 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48196576 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48186904 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚48088628826112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚50552992857600 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚48088645865472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2458314109326 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11029932 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46893767345792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚2799 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29575284 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚48088645865472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144Þî2Ü- _wandbruntime‚1246  _runtime‚1246®n*/Ýü 1loss/SRqqt2/detection_loss‚1.9712086953720462 $loss/total_loss‚35.0334473046408 _runtime‚1246.983978803 _step‚223 0loss/FRt1/assignment_loss‚3.5234840169097437 2loss/SRqqt2/assignment_loss‚1.8578569003895147 1loss/FBt1/assignment_loss‚0.44990831806935605 trainer/global_step‚2799 % _timestamp‚1.7857945729071941e+09 .loss/FBt2/detection_loss‚5.501542018715303 epoch‚0 /loss/FRt2/assignment_loss‚4.548988027572632 2loss/SRqqt1/assignment_loss‚1.4477511821314692 0loss/FBt2/assignment_loss‚0.2796202952890643 .loss/FBt1/detection_loss‚7.782915640048733 /loss/FRt1/detection_loss‚2.6352785943132457 /loss/FRt2/detection_loss‚3.2457348044713337 1loss/SRqqt1/detection_loss‚1.78915881135835658`PÞ /loss/FRt1/detection_loss‚2.6352785943132457 /loss/FRt2/detection_loss‚3.2457348044713337 1loss/SRqqt1/detection_loss‚1.7891588113583565 1loss/SRqqt2/detection_loss‚1.9712086953720462 $loss/total_loss‚35.0334473046408 _runtime‚1246.983978803 _step‚223 0loss/FRt1/assignment_loss‚3.5234840169097437 2loss/SRqqt2/assignment_loss‚1.8578569003895147 1loss/FBt1/assignment_loss‚0.44990831806935605 trainer/global_step‚2799 % _timestamp‚1.7857945729071941e+09 .loss/FBt2/detection_loss‚5.501542018715303 epoch‚0 /loss/FRt2/assignment_loss‚4.548988027572632 2loss/SRqqt1/assignment_loss‚1.4477511821314692 0loss/FBt2/assignment_loss‚0.2796202952890643 .loss/FBt1/detection_loss‚7.782915640048733ßzÒ¦Vß‚(:Ë  ™ ÄÓ™ÎÙ¹ network.recv‚221448 disk.rbd1.in‚0 disk.drbd1289.in‚0 disk.sda.out‚ 33.1015625 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1673.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0# disk.nvme0n1.out‚33.20166015625 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1552.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0* disk./.usagePercent‚31.895731616695322 disk.rbd0.in‚0 disk.drbd1817.in‚0$ proc.memory.rssMB‚11253.33984375 cpu‚13.95304793049737 disk.nvme0n1.in‚0.734375 disk.drbd1550.out‚0 disk.drbd1285.in‚0 disk.sda2.out‚ 33.1015625 disk.drbd1584.in‚0* proc.memory.availableMB‚38526.02734375 network.sent‚ 2.678566e+06 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd1938.in‚0$ memory_percent‚41.21394753456116 disk.drbd2078.in‚0 disk.drbd1965.out‚0 disk.rbd1.out‚ 0.70849609375 disk.sda2.in‚0 disk.drbd1828.in‚0 disk.drbd2041.in‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0* proc.memory.percent‚17.171233892440796 disk.drbd1584.out‚0 disk.drbd1289.out‚0$ disk./.usageGB‚570.3280754089355 disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚0.734375 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.dm-0.out‚33.20166015625 disk.drbd1226.out‚0 proc.cpu.threads‚29 disk.drbd2118.in‚0 disk.sda.in‚0 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.drbd1938.out‚0€iyœà‚(:‘ ™ ÄÓ½©¹ gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚94.484' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚31.494666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åŽ912á- _wandbruntime‚1269  _runtime‚1269е„OÇâÁ trainer/global_step‚2849 % _timestamp‚1.7857945951434808e+09 _runtime‚1269.220472101 _step‚224 & lr-AdamW/pg1‚7.342495929663707e-05 & lr-AdamW/pg2‚7.342495929663707e-05$°><ÌãÆ _step‚224 & lr-AdamW/pg1‚7.342495929663707e-05 & lr-AdamW/pg2‚7.342495929663707e-05 trainer/global_step‚2849 % _timestamp‚1.7857945951434808e+09 _runtime‚1269.220472101ày’î2ä- _wandbruntime‚1269  _runtime‚12691KÂC¼'å¶O PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18937129 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8167682 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49046107214848 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8167949 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚51437176284672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2501325099500 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30092763 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19390527 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚51437280753664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2575411160576 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51621680244736 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚50215275085848 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚47714052828832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2849 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51437176284672 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49039548 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚47714035789472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2507376071168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30092763 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51437176284672 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49039548 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚47714035789472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2507376071168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30092763 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2501325099500 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30092763 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19390527 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚51437280753664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2575411160576á ` ¤2ç- _wandbruntime‚1269  _runtime‚1269Û:&²Æ&èÀM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18943770 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30112956 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49063339235328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚47730793001152 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19397267 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚51455327143424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49047056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11226784 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51639626584064 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2502116769598 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30103286 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚2849 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49056728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7857945955198917e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30112956 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30103286 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19397530 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8170746 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8170485 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚47730775961792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚50232995573986 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2508255690752 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚48946984015872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11226782 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2508343127552 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚48946984015872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49063314331648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2508255690752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2502202572834 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚226 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18943772 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚51455327143424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2576312252416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _runtime‚1269.597214887 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49056728 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0E3Ë&éÅM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30112956 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49063339235328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚47730793001152 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19397267 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚51455327143424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49047056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11226784 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51639626584064 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2502116769598 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30103286 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚2849 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49056728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7857945955198917e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30112956 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30103286 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19397530 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8170746 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8170485 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚47730775961792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚50232995573986 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2508255690752 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚48946984015872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11226782 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2508343127552 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚48946984015872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49063314331648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2508255690752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2502202572834 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚226 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18943772 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚51455327143424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2576312252416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _runtime‚1269.597214887 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49056728 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18943770â”GÚ2ê- _wandbruntime‚1269  _runtime‚1269O€¢ëü 1loss/FBt2/assignment_loss‚0.05350778273008693 /loss/FRt2/detection_loss‚3.2929420875359887 0loss/SRqqt1/detection_loss‚1.383422099144979 1loss/SRqqt2/detection_loss‚3.7586765918348766 _runtime‚1269.598782793 /loss/FBt1/detection_loss‚10.544849192332808 epoch‚0 /loss/FRt1/assignment_loss‚3.134377215968238 2loss/SRqqt1/assignment_loss‚1.6588969009617964 0loss/SRqqt2/assignment_loss‚2.04110895925098 /loss/FRt1/detection_loss‚2.8404150255852274 /loss/FRt2/assignment_loss‚4.424356698989868 1loss/FBt1/assignment_loss‚0.33949082451207296 &loss/total_loss‚35.977196015150824 trainer/global_step‚2849 % _timestamp‚1.7857945955219982e+09 _step‚227 .loss/FBt2/detection_loss‚2.505152636303899+ñC ì .loss/FBt2/detection_loss‚2.505152636303899 1loss/FBt2/assignment_loss‚0.05350778273008693 /loss/FRt2/detection_loss‚3.2929420875359887 0loss/SRqqt1/detection_loss‚1.383422099144979 1loss/SRqqt2/detection_loss‚3.7586765918348766 _runtime‚1269.598782793 /loss/FBt1/detection_loss‚10.544849192332808 epoch‚0 /loss/FRt1/assignment_loss‚3.134377215968238 2loss/SRqqt1/assignment_loss‚1.6588969009617964 0loss/SRqqt2/assignment_loss‚2.04110895925098 /loss/FRt1/detection_loss‚2.8404150255852274 /loss/FRt2/assignment_loss‚4.424356698989868 1loss/FBt1/assignment_loss‚0.33949082451207296 &loss/total_loss‚35.977196015150824 trainer/global_step‚2849 % _timestamp‚1.7857945955219982e+09 _step‚227ã+ƒ†W›í‚(: ¨ ÄÓ‚£÷¸ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚96.511' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚32.17033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06tÚîƒî‚(:Í  ¨ ÄÓœ£¿¹ disk.drbd1285.in‚0$ proc.memory.rssMB‚11253.33984375 disk.drbd1285.out‚0 disk.drbd1356.out‚0) disk./.usagePercent‚31.89575892393895 disk.drbd1552.out‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.sda.out‚ 33.34765625 disk.sda1.in‚0 disk.dm-0.out‚33.70068359375 disk.drbd1938.out‚0 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd1552.in‚0 disk.rbd1.out‚ 0.70849609375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚ 33.34765625 disk.sda1.out‚0 disk.drbd1673.out‚0* proc.memory.availableMB‚38525.73046875 disk.drbd1828.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 proc.cpu.threads‚29* proc.memory.percent‚17.171233892440796 disk.drbd1226.out‚0# disk.nvme0n1.out‚33.70068359375 disk.drbd1550.out‚0 disk.drbd2041.in‚0 disk.drbd1é’Û–Õ931.in‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd2118.in‚0 disk.drbd1356.in‚0 network.sent‚ 2.723544e+06 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0$ memory_percent‚41.21440052986145 disk.drbd2111.out‚0 disk.dm-0.in‚0.734375 disk.drbd1673.in‚0 network.recv‚223711 disk.nvme0n1.in‚0.734375$ disk./.usageGB‚570.3285636901855 disk.drbd1289.out‚0 disk.sda.in‚0 disk.drbd1510.in‚0 cpu‚13.953670443363372 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1817.in‚0>ü{åœï‚(:‘ · ÄÓçÕõ¸ gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚127.874' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚42.62466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Áþ?«Pð‚(:Å  · ÄÓž˜Á¹ disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0( proc.memory.availableMB‚ 38525.703125 proc.cpu.threads‚29 disk.dm-0.in‚0.734375 disk.drbd1938.in‚0 disk.drbd1552.out‚0 cpu‚13.952892490051838$ memory_percent‚41.21444225311279 disk.drbd1356.out‚0 disk.sda.out‚ 33.578125 disk.drbd1938.out‚0) disk./.usagePercent‚31.89581460511541$ disk./.usageGB‚570.3295593261719 disk.drbd1552.in‚0 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.drbd1931.in‚0 disk.rbd0.in‚0 network.recv‚226194 disk.drbd1931.out‚0 disk.drbd1226.out‚0" disk.nvme0n1.out‚ 34.0087890625 disk.drbd2041.in‚0 disk.drbd1356.in‚0 disk.drbd1550.out‚0 disk.sda2.in‚0$ proc.memory.rssMB‚11253.33984375 disk.sda.in‚0 disk.nvme0n1.in‚0.734375 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1510.out‚0 disk.drbd1285.in‚0 disk.drbd1965.in‚0 network.sent‚ 2.768475e+06 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd2118.in‚0 disk.drbd2041.out‚0 disk.dm-0.out‚ 34.0087890625 disk.drbd1817.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2078.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.rbd1.out‚ 0.70849609375 disk.sda2.out‚ 33.578125* proc.memory.percent‚17.171233892440796 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1285.out‚0 disk.drbd1965.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1289.in‚0ùŸ ñ2ñ- _wandbruntime‚1291  _runtime‚1291(,*ðÇòÁ & lr-AdamW/pg1‚7.471356862090237e-05 & lr-AdamW/pg2‚7.471356862090237e-05 trainer/global_step‚2899 % _timestamp‚1.7857946176535468e+09 _runtime‚1291.730548057 _step‚228"«˜ÌóÆ & lr-AdamW/pg1‚7.471356862090237e-05 & lr-AdamW/pg2‚7.471356862090237e-05 trainer/global_step‚2899 % _timestamp‚1.7857946176535468e+09 _runtime‚1291.730548057 _step‚228äe .2ô- _wandbruntime‚1291  _runtime‚12914²~¼'õ¶O KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚228 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49906463054848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 trainer/global_step‚2899 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19269279 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8311677 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2545127759884 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚52339510570496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52339510570496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚51096172165356 _runtime‚1291.731877842 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚49788138363904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19269277 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124382720 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2551459636224 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19731372 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2620719868416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149286400 _step‚229 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30630421 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2551372206592 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49899700 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49906487958528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30620765 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52339615039488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2551459636224 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19269277 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0ÇJÅÁ'ö»O @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚228 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49906463054848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 trainer/global_step‚2899 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19269279 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8311677 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2545127759884 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚52339510570496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52339510570496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚51096172165356 _runtime‚1291.731877842 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚49788138363904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19269277 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124382720 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2551459636224 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19731372 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2620719868416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149286400 _step‚229 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30630421 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2551372206592 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49899700 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49906487958528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30620765 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52339615039488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2551459636224 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19269277åam™R2÷- _wandbruntime‚1292  _runtime‚1292êÎþ˜L#øÀM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚2899 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30631288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49923670171648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49916880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚49805322166272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19275922 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49916880 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30640958 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚52357556953088 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚48567784577792 _runtime‚1292.104787407 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52545330468864 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52357661429248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2552339262976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8314785 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49907208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30640958 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19738168 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30631288 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19275922 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚49805322166272 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19275920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_poo 7XËzl.freed‚19275920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11423632 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8314536 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚48567801617152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚49805305126912ÒË&ùÅM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2546091036342 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2552251826176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _step‚230 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19738419 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49907208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30640958 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19738168 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30631288 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19275922 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚49805322166272 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19275920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19275920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11423632 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8314536 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚48567801617152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚49805305126912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚2899 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30631288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49923670171648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49916880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚49805322166272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19275922 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49916880 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30640958 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚52357556953088 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚48567784577792 _runtime‚1292.104787407 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52545330468864 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52357661429248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2552339262976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8314785 ODeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11616545 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8455206 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20071499 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2595455771648 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20071753 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2665939594752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8454954 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚53432621523968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚49388053021472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51977172087348 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚53241949325312 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚50646476514304 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2589102026516 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53241844856320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚50766843798528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7857946402413993e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50759852 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31158423 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚50646476514304 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2595368342016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19601429 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19601429 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31148767 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53241844856320 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31148767 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2595455771648 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚51977069244864ý7—Ž„ »O WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51977172087348 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚53241949325312 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚50646476514304 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2589102026516 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53241844856320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚50766843798528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7857946402413993e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50759852 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31158423 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚50646476514304 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2595368342016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19601429 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19601429 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31148767 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53241844856320 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31148767 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2595455771648 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚51977069244864 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50750194 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_b5À’3 atch_start/allocation.all.peak‚11887 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚50646493553664 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11616545 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8455206 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20071499 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2595455771648 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20071753 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2665939594752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8454954 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚53432621523968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚49388053021472éBuµA2… - _wandbruntime‚1314  _runtime‚1314ˆqÆ&† ÀM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7857946406127756e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31168960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚53450723316224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2666838457344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚50663643277312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31168960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50784050915328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚2949 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚50663643277312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2596247961600 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50777032 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19608070 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2596335398400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50784026011648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚53259995715072 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2596247961600 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 _runtime‚1314.690031997 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31159290 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19608072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51994789733002 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚53259891238912 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚53450889372672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11620482 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50767360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20078272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19608070 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2666697304576 _step‚234 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚49404810233152 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19608072 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141152768 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚50663660316672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31159290 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20078520 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032^.ÅUË&‡ ÅM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50767360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20078272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19608070 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2666697304576 _step‚234 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚49404810233152 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19608072 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141152768 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚50663660316672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31159290 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20078520 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7857946406127756e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31168960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚53450723316224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2666838457344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚50663643277312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31168960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50784050915328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚2949 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚50663643277312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2596247961600 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50777032 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19608070 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2596335398400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50784026011648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚53259995715072 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2596247961600 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 _runtime‚1314.690031997 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31159290 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19608072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51994789733002 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚53259891238912 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚53450889372672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11620482êÀù?2ˆ - _wandbruntime‚1314  _runtime‚1314ŒÖæE‰ ü .loss/FBt1/detection_loss‚6.599635993268981 _runtime‚1314.690984873 0loss/FRt1/assignment_loss‚3.3817050905970785 /loss/FRt2/assignment_loss‚4.250739391748008 1loss/FBt1/assignment_loss‚0.49607411553870406 1loss/FBt2/assignment_loss‚0.07667037621459236 /loss/FRt1/detection_loss‚3.0812823089299264 1loss/SRqqt1/detection_loss‚1.4350517830092497 %loss/total_loss‚31.26762490094611 trainer/global_step‚2949 _step‚235 .loss/FBt2/detection_loss‚2.889961840853045 epoch‚0 % _timestamp‚1.7857946406140552e+09 2loss/SRqqt1/assignment_loss‚1.6393901636270252 2loss/SRqqt2/assignment_loss‚2.1723899466296035 -loss/FRt2/detection_loss‚3.05447065985048 1loss/SRqqt2/detection_loss‚2.1902532306794193ß7c€Š  epoch‚0 % _timestamp‚1.7857946406140552e+09 2loss/SRqqt1/assignment_loss‚1.6393901636270252 2loss/SRqqt2/assignment_loss‚2.1723899466296035 -loss/FRt2/detection_loss‚3.05447065985048 1loss/SRqqt2/detection_loss‚2.1902532306794193 .loss/FBt1/detection_loss‚6.599635993268981 _runtime‚1314.690984873 0loss/FRt1/assignment_loss‚3.3817050905970785 /loss/FRt2/assignment_loss‚4.250739391748008 1loss/FBt1/assignment_loss‚0.49607411553870406 1loss/FBt2/assignment_loss‚0.07667037621459236 /loss/FRt1/detection_loss‚3.0812823089299264 1loss/SRqqt1/detection_loss‚1.4350517830092497 %loss/total_loss‚31.26762490094611 trainer/global_step‚2949 _step‚235 .loss/FBt2/detection_loss‚2.889961840853045ëÝCñî‘‹ ‚(:† Õ ÄÓ ëú¸ gpu.0.gpu‚23.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚127.041' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚42.347 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ®aêFŒ ‚(:»  Õ ÄÓð¡Á¹ disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.nvme0n1.in‚0.734375 proc.cpu.threads‚29 network.sent‚ 2.81645e+06 disk.rbd1.in‚0 disk.drbd1938.in‚0$ proc.memory.rssMB‚11253.33984375 disk.drbd1931.in‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1550.out‚0 disk.drbd1938.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0) disk./.usagePercent‚31.89581353842621 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.rbd1.out‚ 0.771484375 disk.drbd2078.in‚0% proc.memory.availableMB‚ 38525.625* proc.memory.percent‚17.171233892440796 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd2041.in‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.sda.out‚ 34.4296875 disk.nvme0n1.out‚ 35.1796875 disk.sda2.in‚0 disk.drbd2111.in‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1289.out‚0% memory_percent‚41.214561462402344 disk.dm-0.out‚ 35.1796875 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd1552.out‚0 disk.drbd1289.in‚0 cpu‚13.94597926945433 network.recv‚229523 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1356.out‚0 disk.sda2.out‚ 34.4296875$ disk./.usageGB‚570.3295402526855 disk.drbd1828.in‚0 disk.dm-0.in‚0.734375 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.drbd1285.out‚0 disk.rbd0.out‚0 disk.drbd1285.in‚0 disk.rbd0.in‚0 disk.drbd1550.in‚0íOX”› ‚(: ä ÄÓàŒŠ¹ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚123.85' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.28333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Kjó¦MŽ ‚(:  ä ÄÓêðѹ disk.drbd1673.in‚0 disk.drbd1285.in‚0 disk.drbd1828.in‚0 disk.sda.in‚0 disk.drbd2118.in‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0! disk.nvme0n1.out‚ 35.912109375 disk.rbd0.in‚0 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd1552.out‚0 disk.drbd1965.in‚0 disk.sda2.in‚0 disk.drbd1510.out‚0 disk.drbd1285.out‚0 disk.drbd1673.out‚0 disk.drbd1550.out‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0% proc.memory.availableMB‚ 38525.625$ proc.memory.rssMB‚11253.33984375* proc.memory.percent‚17.171233892440796% memory_percent‚41.214561462402344$ disk./.usageGB‚570.3295021057129 disk.nvme0n1.in‚0.734375 disk.drbd1938.in‚0* disk./.usagePercent‚31.895811405047798 disk.rbd1.out‚ 0.771484375 disk.rbd0.out‚0 disk.drbd2078.in‚0 disk.drbd1938.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.dm-0.out‚ 35.912109375 disk.drbd1817.out‚0 disk.drbd1510.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.sda.out‚ 34.81640625 disk.drbd1552.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 proc.cpu.threads‚29 network.recv‚232078 disk.drbd1356.out‚0 disk.drbd1289.out‚0 disk.drbd1931.out‚0 cpu‚13.947178850264054 disk.drbd1550.in‚0 disk.drbd1289.in‚0 disk.rbd1.in‚0 disk.drbd1584.in‚0 network.sent‚ 2.8614e+06 disk.sda2.out‚ 34.81640625 disk.drbd2118.out‚0 disk.dm-0.in‚0.734375´?ÁÅ2 - _wandbruntime‚1337  _runtime‚1337‡gxà ½ _step‚236 $ lr-AdamW/pg1‚7.7290787269433e-05 $ lr-AdamW/pg2‚7.7290787269433e-05 trainer/global_step‚2999 % _timestamp‚1.7857946633473866e+09 _runtime‚1337.424310245Àpè~È‘  $ lr-AdamW/pg1‚7.7290787269433e-05 $ lr-AdamW/pg2‚7.7290787269433e-05 trainer/global_step‚2999 % _timestamp‚1.7857946633473866e+09 _runtime‚1337.424310245 _step‚236ìK2#2’ - _wandbruntime‚1337  _runtime‚1337]y-=±“ ¶O _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2710969535488 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51610346 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚51627174734848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8597952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚54338144270336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚51627199638528 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚50225061637472 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚54144283611136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31686425 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19933577 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31686425 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50225078676832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51504814664704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2711102306816 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52857966324372 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2632904686900 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.‚›3 small_pool.current‚9656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19933577 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51504814664704 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54144179142144 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51504831704064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51504831704064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2639451907072 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚54144179142144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11813395 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54144179142144 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51504831704064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51504831704064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2639451907072 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚54144179142144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11813395 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚54338144270336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚51627199638528 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚50225061637472 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚54144283611136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31686425 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19933577 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31686425 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50225078676832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51504814664704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2711102306816 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52857966324372 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2632904686900 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0í°ÖX92• - _wandbruntime‚1337  _runtime‚1337+¬æÆ&– ÀM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31687292 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11817332 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141152768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚54162225524736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2640244097024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50241818849152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2640331533824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚51644406755328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 % _timestamp‚1.7857946637133348e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51627512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19940220 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19940222 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚54356254320128 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50241801809792 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31687292 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8600814 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51521981427712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20418148 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19940220 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19940222 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚54356420376576 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20417895 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚51644381851648 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2711872468480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚54162330000896 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31696962 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51521981427712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51521998467072 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2633867963358 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54162330000896 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚52875686812510 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _step‚238 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51637184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11817334 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130ZAÞš— ÅM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2712013621248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19940220 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19940222 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚54356254320128 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50241801809792 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31687292 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8600814 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51521981427712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20418148 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19940220 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19940222 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚54356420376576 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20417895 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚51644381851648 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2711872468480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚54162330000896 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31696962 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51521981427712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51521998467072 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2633867963358 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54162330000896 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚52875686812510 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _step‚238 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51637184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 VIDeviceStatsMonitor.on_train_batch_end/F@|Õ² inactive_split.large_pool.allocated‚11817334 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31687292 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11817332 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141152768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚54162225524736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2640244097024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50241818849152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2640331533824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚51644406755328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 % _timestamp‚1.7857946637133348e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51627512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ODeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51062087292832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚52487530574848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55046617896960 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚52363152815104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32204771 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20750327 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2683448042496 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚55046513427968 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2683448042496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚53738863403880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚3049 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8740082 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55243622098944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52470498 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _step‚241 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2676878953532 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55046513427968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32204771 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚52363169854464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55046617896960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51062087292832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚52487530574848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55046617896960 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚52363152815104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32204771 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20750327 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2683448042496 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚55046513427968 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2683448042496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚53738863403880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚3049 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8740082 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55243622098944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52470498 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _step‚241 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2676878953532 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55046513427968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32204771 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚52363169854464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55046617896960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2677756426866 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52497336 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚52380336617472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2757134413824 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53756583892018 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51078810425792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52497336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52504762595328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51078827465152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3049 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55064664286720 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55064559810560 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55261733049856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32224964 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8742890 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20272372 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚52380336617472 SEDeviceStatsMonitor.on_train_bat™£ch_end/allocated_bytes.small_pool.peak‚ 250523136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20757334 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55064664286720 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚52504737691648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12014184 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8743150 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2684240232448 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚52380319578112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20757072 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52487664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚52380319578112 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _step‚242 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2684240232448 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32215294 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32215294 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20272372 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12014182 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _runtime‚ 1360.828795 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2684327669248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20272370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55261897009152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2677670623630 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52487664 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 % _timestamp‚1.7857946867514613e+09FÄikÈ&¥ ÂM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52487664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚52380319578112 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _step‚242 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2684240232448 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32215294 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32215294 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20272372 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12014182 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _runtime‚ 1360.828795 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2684327669248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20272370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55261897009152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2677670623630 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52487664 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 % _timestamp‚1.7857946867514613e+09 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20272370 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2677756426866 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52497336 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚52380336617472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2757134413824 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53756583892018 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51078810425792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52497336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52504762595328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51078827465152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3049 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55064664286720 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55064559810560 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55261733049856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32224964 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8742890 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20272372 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚52380336617472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20757334 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55064664286720 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚52504737691648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12014184 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8743150 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2684240232448 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚52380319578112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20757072 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ODeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8882519 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2801291055104 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32742429 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20597879 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32732773 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32732773 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚53347886414848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 _step‚245 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚53221508004864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚53221490965504 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32742429 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚53221490965504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2801415437824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.785794708309804e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55948847713792 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55948952182784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124382720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55948952182784<—KÓm² ºO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚53221508004864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚53221490965504 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32742429 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚53221490965504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2801415437824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.785794708309804e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55948847713792 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55948952182784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124382720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55948952182784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2727356748288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚53221508004864 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8882519 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2801291055104 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32742429 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20597879 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32732773 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32732773 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚53347886414848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 _step‚245õ-ðœ 2³ - _wandbruntime‚1382  _runtime‚1382}c`Æ&´ ÀM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚238 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32743296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2728236367872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32752966 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53357488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2802192726528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7857947086783845e+09 _step‚246 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21096288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2721644890374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚53365093531648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2728323804672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2802325490688 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20604522 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚53365118435328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51915836081152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20604520 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21096050 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12211032 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚54637480971526 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _runtime‚1382.755749239 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53238657728512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55966894096384 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53238674767872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32743296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8885018 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20604520 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12211034 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2721559087138 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55966998572544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2728236367872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55966998572544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8885254 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53347816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20604522 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53357488 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136ë8»eË&µ ÅM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20604520 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12211034 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2721559087138 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55966998572544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2728236367872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55966998572544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8885254 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53347816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20604522 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53357488 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚238 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32743296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2728236367872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32752966 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53357488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2802192726528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7857947086783845e+09 _step‚246 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21096288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2721644890374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚53365093531648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2728323804672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2802325490688 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20604522 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚53365118435328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51915836081152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20604520 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21096050 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12211032 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚54637480971526 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _runtime‚1382.755749239 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53238657728512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55966894096384 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53238674767872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32743296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8885018 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024ö±i72¶ - _wandbruntime‚1382  _runtime‚1382TÃÓCþ· ø _step‚247 2loss/SRqqt1/assignment_loss‚1.5209971501377144 0loss/SRqqt2/assignment_loss‚1.86938140737383 /loss/FBt1/assignment_loss‚0.576758632605726 1loss/FBt2/assignment_loss‚0.04249853180253585 .loss/FRt1/detection_loss‚2.671303488810857 $ _timestamp‚1.785794708680226e+09 0loss/SRqqt2/detection_loss‚2.930237143795623 epoch‚0 trainer/global_step‚3099 /loss/FRt1/assignment_loss‚3.190811653269662 /loss/FBt1/detection_loss‚5.2717531811107285 &loss/total_loss‚29.225928038886046 _runtime‚1382.757016908 /loss/FRt2/assignment_loss‚4.354951853162787 /loss/FRt2/detection_loss‚2.6993144743897943 0loss/SRqqt1/detection_loss‚1.274017392713114 /loss/FBt2/detection_loss‚2.8239031297136745QüO[¸ ý /loss/FRt1/assignment_loss‚3.190811653269662 /loss/FBt1/detection_loss‚5.2717531811107285 &loss/total_loss‚29.225928038886046 _runtime‚1382.757016908 /loss/FRt2/assignment_loss‚4.354951853162787 /loss/FRt2/detection_loss‚2.6993144743897943 0loss/SRqqt1/detection_loss‚1.274017392713114 /loss/FBt2/detection_loss‚2.8239031297136745 _step‚247 2loss/SRqqt1/assignment_loss‚1.5209971501377144 0loss/SRqqt2/assignment_loss‚1.86938140737383 /loss/FBt1/assignment_loss‚0.576758632605726 1loss/FBt2/assignment_loss‚0.04249853180253585 .loss/FRt1/detection_loss‚2.671303488810857 $ _timestamp‚1.785794708680226e+09 0loss/SRqqt2/detection_loss‚2.930237143795623 epoch‚0 trainer/global_step‚3099÷¤·‘¹ ‚(:†  ¡ÄÓ‘¯ö¸ gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.514' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.838 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«Â×íVº ‚(:Ë   ¡ÄÓ¾çù disk.drbd1817.in‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚0.734375 cpu‚13.950460972943013 disk.drbd1226.out‚0 disk.drbd1356.out‚0 disk.drbd1289.in‚0 disk.drbd1550.in‚0 disk.drbd1828.in‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd2041.out‚0 proc.cpu.threads‚29 disk.drbd1673.in‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1931.in‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.rbd0.in‚0* proc.memory.availableMB‚38525.53515625$ proc.memory.rssMB‚11253.33984375* proc.memory.percent‚17.171233892440796 disk.drbd1931.out‚0$ memory_percent‚41.21469855308533 disk.drbd1817.out‚0 disk.drbd1552.in‚0 disk.drbd2078.in‚0 disk.drbd1550.out‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 network.sent‚ 2.957445e+06 disk.drbd1510.in‚0 disk.drbd1828.out‚0 disk.drbd2118.in‚0 disk.sda.in‚0) disk./.usagePercent‚31.89581460511541 disk.drbd1673.out‚0 disk.drbd1226.in‚0 disk.drbd1938.in‚0 disk.sda.out‚ 36.65234375$ disk./.usageGB‚570.3295593261719# disk.nvme0n1.out‚37.98388671875 disk.drbd1285.out‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1584.in‚0 disk.dm-0.out‚37.98388671875 disk.rbd1.out‚ 0.771484375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚ 36.65234375 disk.drbd1289.out‚0 disk.sda1.in‚0 network.recv‚239083 disk.nvme0n1.in‚0.734375 disk.drbd1584.out‚0h†á2» - _wandbruntime‚1404  _runtime‚1404_}­Q¼ ¾ % lr-AdamW/pg1‚8.11566152422289e-05 % lr-AdamW/pg2‚8.11566152422289e8ÿs-05 trainer/global_step‚3149 $ _timestamp‚1.785794730536851e+09 _runtime‚1404.613800331 _step‚248Dämɽ à % lr-AdamW/pg1‚8.11566152422289e-05 % lr-AdamW/pg2‚8.11566152422289e-05 trainer/global_step‚3149 $ _timestamp‚1.785794730536851e+09 _runtime‚1404.613800331 _step‚248øô—->2¾ - _wandbruntime‚1404  _runtime‚1404Žõmj¼'¿ ¶O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚54079846155264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21428605 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚57054865377792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2764655880548 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2764570077424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20930029 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚56851286468608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20930029 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚54079829115904 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2771440313344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52736104524832 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚57054699314176 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52736087485472 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚55500657562896 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20930027 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21428865 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 141159936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 166063616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2846457059328 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚55500760405380 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚56851286468608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚54079829115904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20930027 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33260775 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2846598219264 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚56851181999616 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2771352883712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33270431 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚56851181999616 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33270431 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33260775 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚54079846155264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9024660 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12403947 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54190802 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2771352883712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12403945 % _timestamp‚1.7857947305379152e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚54079846155264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21428605 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚57054865377792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2764655880548 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2764570077424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20930029 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚56851286468608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20930029 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚54079829115904 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2771440313344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52736104524832 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚57054699314176 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52736087485472 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚55500657562896 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20930027 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21428865 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 141159936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 166063616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2846457059328 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚55500760405380 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚56851286468608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚54079829115904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20930027 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33260775 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2846598219264 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚56851181999616 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2771352883712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33270431 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚56851181999616 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33270431 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33260775 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚54079846155264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9024660 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12403947 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54190802 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2771352883712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12403945 % _timestamp‚1.7857947305379152e+09ùà"}ø2Á - _wandbruntime‚1404  _runtime‚1404^*ˆÃÅ& ¿M YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52752827657792 _runtime‚1404.980110223 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33271298 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12407884 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54217640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54096995878912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2772319940096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21435382 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚55518378051034 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52752844697152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2772232503296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20936670 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54207968 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20936672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54225449371648 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚56869228382208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54217640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33280968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2772319940096 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚56869332858368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20936672 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2765447550646”>&;æà ÄM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33280968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2772319940096 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚56869332858368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20936672 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2765447550646 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52752827657792 _runtime‚1404.980110223 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33271298 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12407884 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54217640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54096995878912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2772319940096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21435382 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚55518378051034 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52752844697152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2772232503296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20936670 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54207968 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20936672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54225449371648 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚56869228382208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54217640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596ú|»Ö'2Ä - _wandbruntime‚1404  _runtime‚1404…°aèÅ û 1loss/SRqqt2/detection_loss‚2.3025687475421774 .loss/FBt1/detection_loss‚3.681595673927894 epoch‚0 % _timestamp‚1.7857947309044945e+09 0loss/FRt1/assignment_loss‚3.3933514572679995 /loss/FRt2/assignment_loss‚4.255789020393468 2loss/SRqqt1/assignment_loss‚1.5764915303333655 2loss/SRqqt2/assignment_loss‚1.7865540457738411 %loss/total_loss‚27.09695510444179 trainer/global_step‚3149 _step‚251 0loss/FBt1/assignment_loss‚0.4445602183964533 /loss/FRt1/detection_loss‚2.7725786057300867 .loss/FBt2/detection_loss‚2.270078912260039 /loss/FRt2/detection_loss‚2.9843457126164736 0loss/SRqqt1/detection_loss‚1.545226430482153 _runtime‚ 1404.98127534 1loss/FBt2/assignment_loss‚0.08381474971783494ÔŠ,Æ € 1loss/FBt2/assignment_loss‚0.08381474971783494 1loss/SRqqt2/detection_loss‚2.3025687475421774 .loss/FBt1/detection_loss‚3.681595673927894 epoch‚0 % _timestamp‚1.7857947309044945e+09 0loss/FRt1/assignment_loss‚3.3933514572679995 /loss/FRt2/assignment_loss‚4.255789020393468 2loss/SRqqt1/assignment_loss‚1.5764915303333655 2loss/SRqqt2/assignment_loss‚1.7865540457738411 %loss/total_loss‚27.09695510444179 trainer/global_step‚3149 _step‚251 0loss/FBt1/assignment_loss‚0.4445602183964533 /loss/FRt1/detection_loss‚2.7725786057300867 .loss/FBt2/detection_loss‚2.270078912260039 /loss/FRt2/detection_loss‚2.9843457126164736 0loss/SRqqt1/detection_loss‚1.545226430482153 _runtime‚ 1404.98127534ûWÅQ›SÇ ‚(:È  ¯¡ÄÓÁ붹 disk.drbd1289.out‚0 network.recv‚241619 disk.drbd1673.out‚0 disk.drbd1356.out‚0 disk.sda2.out‚ 36.83984375 disk.drbd2078.out‚0 disk.drbd1550.in‚0* disk./.usagePercent‚31.895811405047798 disk.drbd1673.in‚0 disk.drbd1552.out‚0 disk.rbd1.out‚ 0.771484375 disk.rbd0.out‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0# disk.nvme0n1.out‚38.24560546875 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1938.in‚0 disk.sda2.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 36.83984375 proc.cpu.threads‚29 disk.sda1.in‚0 disk.dm-0.in‚0.734375 disk.nvme0n1.in‚0.734375 disk.rbd1.in‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1584.in‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1817.out‚0 disk.drbd1510.out‚0 disk.drbd1931.out‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.rbd0.in‚0$ memory_percent‚41.21427536010742 network.sent‚ 3.002351e+06 disk.drbd1817.in‚0 disk.drbd2111.in‚0$ proc.memory.rssMB‚11253.33984375 disk.drbd1938.out‚0 disk.drbd1226.in‚0 disk.drbd1965.in‚0* proc.memory.percent‚17.171233892440796 disk.drbd1285.out‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 cpu‚13.946649178971779 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.sda.in‚0$ disk./.usageGB‚570.3295021057129 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1285.in‚0& proc.memory.availableMB‚ 38525.8125 disk.dm-0.out‚38.24560546875Q‡ú-œÈ ‚(:‘ ¯¡ÄÓÓ¥ð¹ gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚125.863' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.95433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03žîœÉ ‚(:‘ ¾¡ÄÓäÀþ¸ gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚94.638' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚31.546000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$ÔÁ‹RÊ ‚(:Ç  ¾¡ÄÓƒÀù disk.nvme0n1.in‚0.734375 disk.drbd1552.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1931.out‚0 disk.drbd1510.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚ 37.12890625 disk.drbd1828.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0 disk.drbd1584.out‚0 disk.sda.in‚0! disk.nvme0n1.out‚ 38.685546875 disk.drbd1285.out‚0$ proc.memory.rssMB‚11253.33984375 disk.sda.out‚ 37.12890625 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1817.out‚0 network.recv‚243771* proc.memory.availableMB‚38525.43359375 disk.drbd1289.out‚0 disk.drbd1550.in‚0 disk.drbd1226.in‚0 disk.rbd1.out‚ 0.771484375 proc.cpu.threads‚29 disk.drbd2078.in‚0 disk.drbd1931.in‚0 disk.drbd1510.out‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.drbd1938.in‚0 disk.drbd1673.in‚0 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.sda2.in‚0* disk./.usagePercent‚31.895811405047798 disk.drbd2111.in‚0 disk.dm-0.out‚ 38.685546875 cpu‚13.942574959433207 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.rbd1.in‚0 network.sent‚ 3.04751e+06 disk.sda1.in‚0$ disk./.usageGB‚570.3295021057129 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0* proc.memory.percent‚17.171233892440796$ memory_percent‚41.21485352516174 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.dm-0.in‚0.734375 disk.drbd1938.out‚0Æcin2Ë - _wandbruntime‚1426  _runtime‚1426÷*8ÅÌ ¿ _runtime‚1426.610983596 _step‚252 % lr-AdamW/pg1‚8.24452245664942e-05 % lr-AdamW/pg2‚8.24452245664942e-05 trainer/global_step‚3199 % _timestamp‚1.7857947525340457e+09±©XÊÍ Ä % lr-AdamW/pg1‚8.24452245664942e-05 % lr-AdamW/pg2‚8.24452245664942e-05 trainer/global_step‚3199 % _timestamp‚1.7857947525340457e+09 _runtime‚1426.610983596 _step‚252üZú±2Î - _wandbruntime‚1426  _runtime‚1426:؃œ»'Ï µO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12600797 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2815436448768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33788777 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚57960459395072 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚55068622998528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚253 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55050954 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57753620754432 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33798433 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚56381657484888 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚54938167266304 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9167593 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57753516285440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2808458540932 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 1426.61228096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21262179 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33788777 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 166063616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2891836396544 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚56381554642404 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9167328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21262177 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2815349019136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21768390 % _timestamp‚1.7857947525350642e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55060612 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21768123 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚53573113140832 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12600795 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚54938184305664 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57753620754432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33798433 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 141159936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55068598094848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚53573096101472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2808544344056 trainer/global_step‚3199 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2815349019136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0Ì/"ö Ð ºO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21262179 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33788777 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 166063616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2891836396544 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚56381554642404 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9167328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21262177 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2815349019136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21768390 % _timestamp‚1.7857947525350642e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55060612 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21768123 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚53573113140832 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12600795 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚54938184305664 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57753620754432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33798433 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 141159936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55068598094848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚53573096101472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2808544344056 trainer/global_step‚3199 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2815349019136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12600797 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2815436448768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33788777 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚57960459395072 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚55068622998528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚253 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55050954 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57753620754432 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33798433 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚56381657484888 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚54938167266304 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9167593 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57753516285440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2808458540932 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 1426.61228096ýŠð’2Ñ - _wandbruntime‚1426  _runtime‚1426Èë8ÖÆ&Ò ÀM _runtime‚1426.979480949 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7857947529021149e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚57771667144192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2816228638720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55085805211648 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12604732 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57978408804352 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33808970 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2892603592704 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚53589836273792 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21774877 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2816316075520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21268822 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2809421817390 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12604734 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚53589853313152 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57771667144192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2892732162560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56399275130542 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56399172287946 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57771562668032 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55077792 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚54955351068672 trainer/global_step‚3199 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33808970 _step‚254 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55077792 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9170145 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21268820 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54955351068672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54955334029312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21268822 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12604732 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57978408804352 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33808970 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2892603592704 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚53589836273792 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21774877 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2816316075520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21268822 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2809421817390 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12604734 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚53589853313152 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57771667144192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2892732162560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56399275130542 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56399172287946 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57771562668032 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55077792 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚54955351068672 trainer/global_step‚3199 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33808970 _step‚254 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55077792 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9170145 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21268820 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54955351068672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54955334029312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21268822 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7857947529021149e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚57771667144192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2816228638720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55085805211648 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚58655955040256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12797645 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2859432584192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2859345154560 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9309394 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2852432807564 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21594327 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55796522456064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34326435 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚55796522456064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚58655955040256 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2859432584192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚3249 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21594329 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚57262451721912 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9309650 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21594327 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34316779 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54410121756832 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚55796505416704 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 % _timestamp‚1.7857947747275198e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12797647 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 @8DeviceStatsMonitor.on_train_batch_startPÆ /segment.all.peak‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34326435 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58865989408768 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚58655850571264 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2937010570240 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚58655850571264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚55796505416704 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2859345154560 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 % _timestamp‚1.7857947747275198e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12797647 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34326435 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58865989408768 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚58655850571264 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2937010570240 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚58655850571264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚55796505416704 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2859345154560 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚58655955040256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640ÄÆbê2ß - _wandbruntime‚1449  _runtime‚1449áR:Å&à ¿M E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55937944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54426844889792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2853310280898 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2853224477662 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚57280069367454 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55937944 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58883947997184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2860224774144 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚58673896953856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55928272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2860312210944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34327302 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55946185955328 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22114041 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12801584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 _runtime‚1449.172081574 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21600972 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2937786945536 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21600970 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚58674001430016 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55928272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22113795 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚55813689219072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34327302 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚57280172210050 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.785794775094567e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9312457 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12801582 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34336972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21600972 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34336972 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.785794775094567e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9312457 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12801582 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34336972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21600972 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34336972 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55937944 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58883947997184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2860224774144 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚58673896953856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55928272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2860312210944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34327302 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55946185955328 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22114041 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12801584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 _runtime‚1449.172081574 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21600972 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2937786945536 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21600970 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚58674001430016 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55928272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22113795 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚55813689219072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitorˆˆDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34327302 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0‚€PŸæ2â - _wandbruntime‚1449  _runtime‚1449céé«ã ú /loss/FRt2/detection_loss‚2.7190254469250523 trainer/global_step‚3249 $ _timestamp‚1.785794775096469e+09 0loss/FRt2/assignment_loss‚3.9087587789047595 /loss/FRt1/detection_loss‚3.1747870113867434 1loss/SRqqt2/detection_loss‚5.0383803941805825 /loss/FRt1/assignment_loss‚3.139104449926917 1loss/SRqqt1/assignment_loss‚1.639097806636025 1loss/SRqqt1/detection_loss‚1.1080243800753071 .loss/FBt1/detection_loss‚3.680217025180658 .loss/FBt2/detection_loss‚2.070782889085487 _runtime‚1449.173232122 _step‚259 1loss/SRqqt2/assignment_loss‚1.615628267035765 epoch‚0 %loss/total_loss‚28.70734544954276 0loss/FBt1/assignment_loss‚0.5238151674469312 1loss/FBt2/assignment_loss‚0.08972383275853314¯`eä ÿ %loss/total_loss‚28.70734544954276 0loss/FBt1/assignment_loss‚0.5238151674469312 1loss/FBt2/assignment_loss‚0.08972383275853314 /loss/FRt2/detection_loss‚2.7190254469250523 trainer/global_step‚3249 $ _timestamp‚1.785794775096469e+09 0loss/FRt2/assignment_loss‚3.9087587789047595 /loss/FRt1/detection_loss‚3.1747870113867434 1loss/SRqqt2/detection_loss‚5.0383803941805825 /loss/FRt1/assignment_loss‚3.139104449926917 1loss/SRqqt1/assignment_loss‚1.639097806636025 1loss/SRqqt1/detection_loss‚1.1080243800753071 .loss/FBt1/detection_loss‚3.680217025180658 .loss/FBt2/detection_loss‚2.070782889085487 _runtime‚1449.173232122 _step‚259 1loss/SRqqt2/assignment_loss‚1.615628267035765 epoch‚0ƒÁ8¯‘å ‚(:† Ü¡ÄÓé•Ÿ¹ gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚122.823' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.941 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ²}KSæ ‚(:È  Ü¡ÄÓ–Žê¹ network.recv‚247049 disk.drbd1226.in‚0 disk.nvme0n1.in‚0.734375 disk.drbd1965.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1510.out‚0 disk.drbd1226.out‚0 disk.drbd1285.in‚0 disk.sda2.out‚ 37.8828125 disk.drbd1584.out‚0% memory_percent‚41.215938329696655 disk.drbd1356.out‚0 disk.sda2.in‚0 disk.drbd1931.in‚0 cpu‚13.934583269087412$ disk./.usageGB‚570.3301849365234 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1828.in‚0 disk.sda.out‚ 37.8828125 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.sda.in‚0 disk.drbd2041.out‚0 disk.sda1.out‚0 disk.drbd1356.in‚0 disk.drbd1552.in‚0* proc.memory.availableMB‚38524.72265625 disk.drbd1552.out‚0 disk.drbd2118.out‚0$ proc.memory.rssMB‚11253.33984375 disk.drbd1285.out‚0 disk.rbd0.out‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1673.in‚0 network.sent‚ 3.095483e+06) disk./.usagePercent‚31.89584959252131 disk.drbd2111.in‚0" disk.nvme0n1.out‚ 39.4072265625 disk.rbd0.in‚0 disk.drbd1828.out‚0 disk.drbd1584.in‚0 disk.sda1.in‚0 disk.dm-0.out‚ 39.4072265625 disk.drbd1510.in‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0* proc.memory.percent‚17.171233892440796 disk.rbd1.out‚ 0.771484375 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.drbd1817.in‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.drbd1550.out‚0 disk.drbd2041.in‚0 disk.dm-0.in‚0.734375 proc.cpu.threads‚29b‘sUç ‚(:Ê  ë¡ÄÓ®íè¹ disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.drbd2111.out‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.sda2.in‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1356.out‚0 disk.dm-0.in‚0.734375 disk.drbd1552.in‚0$ memory_percent‚41.21555685997009 disk.drbd1817.out‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.drbd1510.in‚0 disk.sda2.out‚ 38.16796875 disk.drbd1828.in‚0* proc.memory.percent‚17.171233892440796 disk.drbd1226.in‚0 disk.nvme0n1.in‚0.734375 disk.drbd1356.in‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.sda.out‚ 38.16796875 disk.sda1.out‚0 disk.drbd1510.out‚0 proc.cpu.threads‚29 disk.drbd1931.in‚0 disk.drbd1938.in‚0" disk.nvme0n1.out‚ 39.5517578125 disk.drbd1285.in‚0 disk.rbd0.out‚0$ proc.memory.rssMB‚11253.33984375 cpu‚13.930748212113494* disk./.usagePercent‚31.895855992656536 disk.rbd0.in‚0 disk.drbd2078.out‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd1289.in‚0 disk.drbd2111.in‚0* proc.memory.availableMB‚38524.97265625 network.recv‚249397 disk.drbd1552.out‚0 disk.rbd1.out‚ 0.771484375 disk.drbd2118.in‚0 disk.sda.in‚0 disk.drbd1584.out‚0 disk.drbd1285.out‚0 network.sent‚ 3.140399e+06$ disk./.usageGB‚570.3302993774414 disk.dm-0.out‚ 39.5517578125½DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2896321271072 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56789334678528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21926477 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59558184857088 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56780916 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9452974 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59558184857088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21926479 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚56654843567104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22447471 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚56654843567104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 % _timestamp‚1.7857947970999153e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚56654860606464 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56771258 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34844781 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12994495 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚55247113333472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2903428719616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21926479 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34844781 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12994497 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55247130372832 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 _runtime‚1471.180636442 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34854437 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59558184857088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21926479 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚56654843567104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22447471 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚56654843567104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 % _timestamp‚1.7857947970999153e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚56654860606464 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56771258 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34844781 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12994495 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚55247113333472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2903428719616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21926479 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34844781 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12994497 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55247130372832 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 _runtime‚1471.180636442 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34854437 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2896321271072 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56789334678528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21926477 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59558184857088 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56780916 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9452974 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56798096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2897112941170 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34855304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚56672010330112 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21933122 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21933120 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34864974 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9455760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚58160966446962 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2904220909568 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 C6DeviceStL'ƘTatsMonitor.on_train_batch_end/active.all.freed‚56788424 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚56672010330112 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22454194 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34855304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56788424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56798096 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12998434 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2897198744406 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚262 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 _runtime‚1471.555248544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7857947974778767e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21933120 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22453944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2904308346368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚3299¿ÆI?Ë&ñ ÅM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59576335715840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55263853505792 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚56806541795328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56798096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2897112941170 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34855304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚56672010330112 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21933122 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21933120 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34864974 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9455760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚58160966446962 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2904220909568 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56788424 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚56672010330112 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22454194 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34855304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56788424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56798096 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12998434 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2897198744406 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚262 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 _runtime‚1471.555248544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7857947974778767e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21933120 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22453944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2904308346368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚3299 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58161069289558 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12998432 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0†]½2ò - _wandbruntime‚1471  _runtime‚1471€¿‘ó ý $ _timestamp‚1.785794797480089e+09 1loss/SRqqt2/assignment_loss‚1.596977762554003 /loss/FRt1/detection_loss‚2.9661110290459223 .loss/FBt1/detection_loss‚4.029255969198253 _runtime‚1471.556911742 epoch‚0 2loss/SRqqt1/assignment_loss‚1.6489401242046646 1loss/SRqqt1/detection_loss‚1.2660686627806474 /loss/FBt2/detection_loss‚2.5999931825628115 1loss/FBt1/assignment_loss‚0.26795935854482067 0loss/FBt2/assignment_loss‚0.0536720724223076 trainer/global_step‚3299 _step‚263 0loss/FRt1/assignment_loss‚3.2240288308688574 0loss/FRt2/assignment_loss‚4.0815519210768905 .loss/FRt2/detection_loss‚2.769452467924211 1loss/SRqqt2/detection_loss‚3.6030506758910157 &loss/total_loss‚28.107062057074405Sar‘ô ‚ $ _timestamp‚1.785794797480089e+09 1loss/SRqqt2/assignment_loss‚1.596977762554003 /loss/FRt1/detection_loss‚2.9661110290459223 .loss/FBt1/detection_loss‚4.029255969198253 _runtime‚1471.556911742 epoch‚0 2loss/SRqqt1/assignment_loss‚1.6489401242046646 1loss/SRqqt1/detection_loss‚1.2660686627806474 /loss/FBt2/detection_loss‚2.5999931825628115 1loss/FBt1/assignment_loss‚0.26795935854482067 0loss/FBt2/assignment_loss‚0.0536720724223076 trainer/global_step‚3299 _step‚263 0loss/FRt1/assignment_loss‚3.2240288308688574 0loss/FRt2/assignment_loss‚4.0815519210768905 .loss/FRt2/detection_loss‚2.769452467924211 1loss/SRqqt2/detection_loss‚3.6030506758910157 &loss/total_loss‚28.107062057074405‡2X šõ ‚(: ú¡ÄÓ¨Äõ¸ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚94.25' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚31.416666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùM2PTö ‚(:É  ú¡ÄÓ­¬»¹ disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd1965.out‚0) disk./.usagePercent‚31.89585492596733 disk.drbd1226.in‚0 disk.sda2.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0$ disk./.usageGB‚570.3302803039551 disk.dm-0.in‚0.734375 disk.drbd1673.in‚0 disk.drbd1510.in‚0 disk.rbd0.out‚0 disk.drbd1289.in‚0 disk.nvme0n1.in‚0.734375" disk.nvme0n1.out‚ 39.9189453125 disk.sda2.out‚ 38.55078125 disk.drbd1550.out‚0 disk.drbd1584.out‚0$ proc.memory.rssMB‚11253.33984375 network.sent‚ 3.143417e+06* proc.memory.percent‚17.171233892440796 disk.drbd1817.out‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd1828.in‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd1965.in‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.rbd1.in‚0 disk.drbd1828.out‚0$ memory_percent‚41.21561646461487 disk.sda.out‚ 38.55078125 disk.sda1.in‚0 disk.rbd1.out‚ 0.771484375 network.recv‚250431 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.dm-0.out‚ 39.9189453125 disk.drbd1289.out‚0 proc.cpu.threads‚29 disk.drbd1931.out‚0 disk.drbd1673.out‚0 disk.rbd0.in‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.sda.in‚0 disk.drbd1817.in‚0 disk.drbd1356.out‚0 disk.drbd1938.out‚0 disk.drbd1584.in‚0* proc.memory.availableMB‚38524.93359375 disk.drbd1285.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1938.in‚0 cpu‚13.926913222894777§T2÷ - _wandbruntime‚1492  _runtime‚1492²ŽÇø Á trainer/global_step‚3349 % _timestamp‚1.7857948188597274e+09 _runtime‚1492.936675212 _step‚264 & lr-AdamW/pg1‚8.631105253929012e-05 & lr-AdamW/pg2‚8.631105253929012e-05ݸMÌù Æ _step‚264 & lr-AdamW/pg1‚8.631105253929012e-05 & lr-AdamW/pg2‚8.631105253929012e-05 trainer/global_step‚3349 % _timestamp‚1.7857948188597274e+09 _runtime‚1492.936675212ˆ Sg2ú - _wandbruntime‚1492  _runtime‚1492€Ôø«¼'û ¶O D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2940123931456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚57649690518528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 trainer/global_step‚3349 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22258629 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2947337425408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56084121949472 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35372783 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59024348723412 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22258629 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚60676900463104 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2940209734580 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚60677062332416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35372783 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35382439 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22258627 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57641068 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60460519142912 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _step‚265 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59024245880928 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56084138988832 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13191345 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2947337425408 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚60460519142912 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3027234848256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60460519142912 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _step‚265 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59024245880928 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56084138988832 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13191345 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2947337425408 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚60460519142912 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3027234848256 ZLDeviceStatsMonitor.on_train_batch_staŒÍ Žrt/inactive_split_bytes.small_pool.peak‚ 145651200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2940123931456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚57649690518528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 trainer/global_step‚3349 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22258629 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2947337425408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56084121949472 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35372783 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59024348723412 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0‰&™o2ý - _wandbruntime‚1493  _runtime‚1493£N-2Å&þ ¿M D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚60478565525504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2941087207914 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22265270 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3028136241152 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59041863526470 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22793555 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚57530365519872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9598273 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57648576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13195282 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56100862121792 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56100879161152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22265270 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57658248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35392976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57648576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2948304481792 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22265272 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚57666897635328 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚60478670001664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57658248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3349 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60478670001664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚57530365519872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 _runtime‚1493.313365054 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35383306 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚57530348480512 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚60695166640640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2941001404678 $ _timestamp‚1.785794819235979e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2948217044992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13195284 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60695008972800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2948304481792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35392976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚57666872731648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9598528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22265272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22793812 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251D#rÊ&ÿ ÄM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚60478565525504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2941087207914 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22265270 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3028136241152 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59041863526470 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22793555 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚57530365519872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9598273 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57648576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13195282 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56100862121792 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56100879161152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22265270 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57658248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35392976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57648576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2948304481792 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22265272 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚57666897635328 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚60478670001664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57658248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3349 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60478670001664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚57530365519872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 _runtime‚1493.313365054 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35383306 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚57530348480512 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚60695166640640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2941001404678 $ _timestamp‚1.785794819235979e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2948217044992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13195284 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60695008972800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2948304481792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35392976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚57666872731648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9598528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22265272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22793812 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130Š1AQ*2€ - _wandbruntime‚1493  _runtime‚1493Î`&" ü /loss/FRt1/assignment_loss‚3.421345848195693 2loss/SRqqt2/assignment_loss‚1.8556092862899487 .loss/FRt2/detection_loss‚2.894243860244751 0loss/FBt1/assignment_loss‚0.5089932109229267 .loss/FBt2/detection_loss‚3.726257644653912 %loss/total_loss‚35.71077195862499 trainer/global_step‚3349 _step‚267 2loss/FBt2/assignment_loss‚0.014947110958473786 .loss/FRt1/detection_loss‚2.608731066830018 1loss/SRqqt1/detection_loss‚1.3504024672572716 /loss/FBt1/detection_loss‚11.165443921089173 _runtime‚1493.314926132 /loss/FRt2/assignment_loss‚4.220590937137604 2loss/SRqqt1/assignment_loss‚1.6908810788240187 1loss/SRqqt2/detection_loss‚2.2533255262211984 epoch‚0 % _timestamp‚1.7857948192381473e+09®¯~O‚  1loss/SRqqt2/detection_loss‚2.2533255262211984 epoch‚0 % _timestamp‚1.7857948192381473e+09 /loss/FRt1/assignment_loss‚3.421345848195693 2loss/SRqqt2/assignment_loss‚1.8556092862899487 .loss/FRt2/detection_loss‚2.894243860244751 0loss/FBt1/assignment_loss‚0.5089932109229267 .loss/FBt2/detection_loss‚3.726257644653912 %loss/total_loss‚35.71077195862499 trainer/global_step‚3349 _step‚267 2loss/FBt2/assignment_loss‚0.014947110958473786 .loss/FRt1/detection_loss‚2.608731066830018 1loss/SRqqt1/detection_loss‚1.3504024672572716 /loss/FBt1/detection_loss‚11.165443921089173 _runtime‚1493.314926132 /loss/FRt2/assignment_loss‚4.220590937137604 2loss/SRqqt1/assignment_loss‚1.6908810788240187‹À¤Ä›ƒ ‚(: ‰¢ÄӼɹ gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚122.46' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.81999999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÚO„ ‚(:Ä  ‰¢Äӣﶺ disk.sda2.out‚ 38.9296875 disk.drbd1550.out‚0 disk.dm-0.out‚ 40.1572265625 disk.drbd1817.in‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1828.in‚0 disk.sda.out‚ 38.9296875$ disk./.usageGB‚570.3302803039551 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.rbd1.out‚ 0.771484375 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0" memory_percent‚41.215980052948 disk.drbd1510.out‚0 proc.cpu.threads‚29 disk.drbd1552.in‚0 disk.drbd1552.out‚0 network.recv‚252965 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.dm-0.in‚0.734375 disk.drbd1817.out‚0 network.sent‚ 3.188393e+06) disk./.usagePercent‚31.89585492596733 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd1965.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚0.734375$ proc.memory.rssMB‚11253.33984375 disk.drbd2111.out‚0 disk.drbd2041.in‚0 disk.drbd1673.in‚0 disk.drbd1356.out‚0 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1673.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1226.in‚0 disk.drbd1965.in‚0) proc.memory.availableMB‚ 38524.6953125* proc.memory.percent‚17.171233892440796 cpu‚13.923366457957048 disk.sda.in‚0 disk.sda1.in‚0" disk.nvme0n1.out‚ 40.1572265625áBWÀV… ‚(:Ë  ˜¢ÄÓ¦’ѹ disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd2118.in‚0 disk.rbd0.in‚0 disk.drbd1510.in‚0 disk.drbd1285.in‚0 proc.cpu.threads‚29 disk.sda.out‚ 39.20703125 disk.dm-0.out‚40.37255859375 disk.drbd1356.in‚0 network.sent‚ 3.233319e+06 disk.rbd0.out‚0 disk.drbd1584.in‚0 disk.drbd1226.in‚0* proc.memory.availableMB‚38525.18359375 disk.sda2.in‚0 disk.sda2.out‚ 39.20703125 disk.drbd1828.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0* proc.memory.percent‚17.171233892440796) disk./.usagePercent‚31.89585492596733 disk.drbd1552.in‚0 disk.rbd1.out‚ 0.771484375 disk.drbd2041.in‚0 disk.dm-0.in‚0.734375 disk.drbd1938.out‚0 disk.drbd1817.in‚0 disk.drbd1285.out‚0 disk.drbd1356.out‚0% memory_percent‚41.215234994888306$ disk./.usageGB‚570.3302803039551 disk.sda1.out‚0# disk.nvme0n1.out‚40.37255859375 disk.drbd1673.in‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd1552.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1673.out‚0 disk.drbd1550.out‚0 disk.sda.in‚0 disk.nvme0n1.in‚0.734375 disk.drbd1817.out‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 network.recv‚255417 disk.sda1.in‚0$ proc.memory.rssMB‚11253.33984375 disk.drbd1931.in‚0 disk.drbd1226.out‚0 disk.drbd1289.in‚0 disk.drbd2111.out‚0 cpu‚13.91984036515838 disk.drbd2078.in‚0 disk.drbd2041.out‚0Rä*† ‚(:’ ˜¢ÄÓˆº gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚115.844' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.614666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£5º¦2‡ - _wandbruntime‚1515  _runtime‚1515šÐH¯Çˆ Á & lr-AdamW/pg1‚8.759966186355543e-05 & lr-AdamW/pg2‚8.759966186355543e-05 trainer/global_step‚3399 % _timestamp‚1.7857948412055209e+09 _runtime‚1515.282532924 _step‚268‰·rñ̉ Æ & lr-AdamW/pg1‚8.759966186355543e-05 & lr-AdamW/pg2‚8.759966186355543e-05 trainer/global_step‚3399 % _timestamp‚1.7857948412055209e+09 _runtime‚1515.282532924 _step‚268Œ!ÁöØ2Š - _wandbruntime‚1515  _runtime‚1515óqS§‹ µO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22590777 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚61582564283904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22590779 L=DeviceStatsMonitor.on_trî;®ƒ&ain_batch_start/active_bytes.all.peak‚ 2940455936 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 trainer/global_step‚3399 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9738297 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35910441 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61362853428736 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58371536907264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56921147604832 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58510046358528 % _timestamp‚1.7857948412066317e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3072517925376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2991420990464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61362853428736 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚61582406608896 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35900785 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58371519867904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2991333560832 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚61362957897728 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58491562 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22590779 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2991420990464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _runtime‚ 1515.28382363 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58371536907264 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35900785 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2984012394964 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2991333560832 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59905142960436 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56921130565472 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35910441 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22590777 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61362957897728 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59905245802920 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 _step‚269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0ŽÁt¸À'Œ ºO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58371536907264 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35900785 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2984012394964 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2991333560832 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59905142960436 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56921130565472 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35910441 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22590777 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61362957897728 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59905245802920 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 _step‚269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22590777 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚61582564283904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22590779 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 trainer/global_step‚3399 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9738297 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35910441 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61362853428736 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58371536907264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56921147604832 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58510046358528 % _timestamp‚1.7857948412066317e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3072517925376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2991420990464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61362853428736 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚61582406608896 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35900785 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58371519867904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2991333560832 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚61362957897728 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58491562 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22590779 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2991420990464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _runtime‚ 1515.28382363ÚœÂ2 - _wandbruntime‚1515  _runtime‚1515hÐYÅ&Ž ¿M REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13392132 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58527253475328 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35920978 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚61600516298240 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3073418393600 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚61600671868928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚61381004287488 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58508728 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59922863448574 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58508728 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9741119 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3073287726592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58518400 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22597420 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58527228571648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22597422 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23133251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35911308 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2992213180416 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚58388703670272 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56937870737792 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61380899811328 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚58388686630912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚270 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2984889868186 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22597422 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2984975671422 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35911308 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 trainer/global_step‚3399 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚61381004287488 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58518400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 $ _timestamp‚1.785794841578739e+09 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23133506 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56937887777152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251_Ñ~×  ÄM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2992300617216 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2992213180416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35920978 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚61600516298240 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3073418393600 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚61600671868928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚61381004287488 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58508728 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59922863448574 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58508728 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9741119 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3073287726592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58518400 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22597420 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58527228571648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.onªÁH®¸_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22597422 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23133251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35911308 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2992213180416 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚58388703670272 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56937870737792 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61380899811328 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚58388686630912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚270 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2984889868186 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22597422 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2984975671422 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35911308 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 trainer/global_step‚3399 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚61381004287488 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58518400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 $ _timestamp‚1.785794841578739e+09 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23133506 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56937887777152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13392132 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58527253475328 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚62265187714560 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚60786142882428 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3035417125888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚59370402198528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62265187714560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚59229858018304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22922927 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22922929 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚59229858018304 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚62265292183552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59361372 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23466592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚62265187714560 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚60786142882428 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3035417125888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚59370402198528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62265187714560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚59229858018304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22922927 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22922929 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚59229858018304 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚62265292183552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59361372 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23466592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚60803760528082 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36448980 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3036209315840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚60803657685486 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22929570 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 K>DeviceStatsMonitor.on_train_batch_end/inactive_splÓ ¥"it.all.freed‚23473108 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22929572 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9884126 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7857948634864438e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62283338573312 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 trainer/global_step‚3449 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚62283338573312 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3028778331694 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _runtime‚1537.563792541 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚62506049282048 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚59387584411648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 _step‚274 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22929572 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59247024781312 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3036296752640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚57774896393152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59378552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59368880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23473108 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22929572 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9884126 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7857948634864438e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62283338573312 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 trainer/global_step‚3449 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚62283338573312 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3028778331694 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _runtime‚1537.563792541 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚62506049282048 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚59387584411648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 _step‚274 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22929572 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59247024781312 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3036296752640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚57774896393152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59378552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59368880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1’Œßˆâ2ž - _wandbruntime‚1537  _runtime‚1537 AjŸ ÿ 2loss/FBt2/assignment_loss‚0.019833321373997962 /loss/FRt2/detection_loss‚2.8538828416715694 epoch‚0 % _timestamp‚1.7857948634882035e+09 _step‚275 0loss/FRt1/assignment_loss‚3.3584472992840935 0loss/FRt2/assignment_loss‚4.1236827267876155 1loss/SRqqt1/detection_loss‚1.2703487384711103 /loss/FBt1/detection_loss‚2.4762902524736288 trainer/global_step‚3449 _runtime‚1537.564978598 2loss/SRqqt2/assignment_loss‚1.8493145082307898 0loss/FBt1/assignment_loss‚0.5127667122417026 -loss/FRt1/detection_loss‚2.92542270088897 /loss/FBt2/detection_loss‚2.6805256495647174 2loss/SRqqt1/assignment_loss‚1.5558330150513813 1loss/SRqqt2/detection_loss‚3.2863032946606046 %loss/total_loss‚26.91265106070018ùE³   „ 1loss/SRqqt2/detection_loss‚3.2863032946606046 %loss/total_loss‚26.91265106070018 2loss/FBt2/assignment_loss‚0.019833321373997962 /loss/FRt2/detection_loss‚2.8538828416715694 epoch‚0 % _timestamp‚1.7857948634882035e+09 _step‚275 0loss/FRt1/assignment_loss‚3.3584472992840935 0loss/FRt2/assignment_loss‚4.1236827267876155 1loss/SRqqt1/detection_loss‚1.2703487384711103 /loss/FBt1/detection_loss‚2.4762902524736288 trainer/global_step‚3449 _runtime‚1537.564978598 2loss/SRqqt2/assignment_loss‚1.8493145082307898 0loss/FBt1/assignment_loss‚0.5127667122417026 -loss/FRt1/detection_loss‚2.92542270088897 /loss/FBt2/detection_loss‚2.6805256495647174 2loss/SRqqt1/assignment_loss‚1.5558330150513813“[Žõ ¡ ‚(:… ¶¢ÄÓ†ë¹ gpu.0.gpu‚8.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚115.383' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.461 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03—?S¢ ‚(:È  ¶¢ÄÓ¢“Ò¹ disk.rbd1.out‚ 0.771484375 cpu‚13.913188825729334 disk.drbd2118.in‚0$ proc.memory.rssMB‚11253.34765625 disk.drbd1289.in‚0 disk.drbd1584.in‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚0.734375 disk.rbd1.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1938.in‚0$ memory_percent‚41.21622443199158 disk.drbd2111.in‚0 disk.sda1.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1356.in‚0 network.sent‚ 3.281396e+06 disk.drbd2041.out‚0 disk.drbd1673.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚0.734375) proc.memory.percent‚17.17124581336975) disk./.usagePercent‚31.89585492596733 disk.drbd2078.in‚0 disk.drbd1550.in‚0 disk.sda.out‚ 40.08203125 disk.drbd1510.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.sda2.out‚ 40.08203125 disk.sda.in‚0 disk.drbd1828.in‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0 disk.dm-0.out‚ 40.9814453125 disk.drbd1938.out‚0* proc.memory.availableMB‚38524.53515625 disk.drbd1584.out‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1673.out‚0 disk.drbd1552.in‚0 disk.drbd2111.out‚0 disk.drbd1817.in‚0" disk.nvme0n1.out‚ 40.9814453125 disk.drbd1552.out‚0 disk.sda2.in‚0 proc.cpu.threads‚29$ disk./.usageGB‚570.3302803039551 disk.drbd1289.out‚0 disk.drbd1931.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1965.in‚0 network.recv‚258561 disk.drbd1817.out‚0èR°2£ - _wandbruntime‚1559  _runtime‚1559é¨Ç¤ Á & lr-AdamW/pg1‚9.017688051208604e-05 & lr-AdamW/pg2‚9.017688051208604e-05 trainer/global_step‚3499 % _timestamp‚1.7857948850554285e+09 _runtime‚1559.132405181 _step‚276t„ÈÌ¥ Æ % _timestamp‚1.7857948850554285e+09 _runtime‚1559.132405181 _step‚276 & lr-AdamW/pg1‚9.017688051208604e-05 & lr-AdamW/pg2‚9.017688051208604e-05 trainer/global_step‚3499”tùo2¦ - _wandbruntime‚1559  _runtime‚1559æ¤ú)&§ ·O VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚61666937119452 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3071789321980 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10024597 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚60230758038528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23806494 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7857948850565355e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚60088196168704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13781897 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23255079 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36956789 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚60088213208064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36966445 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚63167626469376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚63167522000384 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36966445 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60211866 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23255077 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3071875125104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23255077 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58595147797472 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚63167626469376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60230733134848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚61667039961936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3499 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚63167522000384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 ZLDeviceStatsMonitor.»¶{²”on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3079325831680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10024352‡,Â'¨ ¼O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10024597 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚60230758038528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23806494 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7857948850565355e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚60088196168704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13781897 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23255079 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36956789 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚60088213208064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36966445 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚63167626469376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚63167522000384 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36966445 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60211866 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23255077 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3071875125104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23255077 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58595147797472 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚63167626469376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60230733134848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚61667039961936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3499 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚63167522000384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3079325831680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10024352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚61666937119452 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3071789321980 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0•‘¸tÔ2© - _wandbruntime‚1559  _runtime‚1559aˆö@È&ª ÂM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36976982 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63411523530752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23261720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61684657607590 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23812948 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60247965155328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚63185672859136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3080205451264 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60238704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36967312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23813198 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3080205451264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23261722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23261722 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3080292888064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10027364 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60229032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58611905009152 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60238704 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚63185568382976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10027116 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚63185672859136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63411687490048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚60105362931712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3080292888064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60247940251648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58611905009152 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60238704 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚63185568382976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10027116 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚63185672859136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63411687490048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚60105362931712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3080292888064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60247940251648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23261720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61684657607590 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23812948 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60247965155328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚63185672859136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3080205451264 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60238704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36967312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23813198 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3080205451264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23261722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23261722 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3080292888064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10027364 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60229032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13978747 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚61091113878528 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64069856286208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3123321967104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23587229 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37484791 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3115763588612 trainer/global_step‚3549 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3123321967104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24146628 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚64069856286208 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚60946534319104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3207950965248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23587227 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23587227 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3207816096768 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚64069960755200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚62547937041444 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61072018 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚60946551358464 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37494447 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚59432156413472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640|,Ã'¸ ½O C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚61091113878528 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64069856286208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3123321967104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23587229 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37484791 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3115763588612 trainer/global_step‚3549 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3123321967104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24146628 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚64069856286208 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚60946534319104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3207950965248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23587227 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23587227 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3207816096768 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚64069960755200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚62547937041444 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61072018 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚60946551358464 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37494447 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚59432156413472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13978745 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10167881 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚1581.388768623 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61091088974848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3115677785488 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23587229 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10167626 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64298905071616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚281 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37494447 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚64069960755200 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3123409396736 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13978747 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032™rÊ´Â2¹ - _wandbruntime‚1581  _runtime‚1581]Ò@Q º ÂM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚60963718121472 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37504984 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64317010165248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚60963701082112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24153140 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37504984 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61098856 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59448913625152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3124289023488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61108296091648 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚64088007144960 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3124201586688 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59448896585792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3124201586688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚282 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61108320995328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 % _timestamp‚1.7857949076787152e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61098856 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚640Cxðû+88007144960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3124289023488 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61089184 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64317176221696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚62565554687098 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23593870 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3116555258710 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032ÏO³Í&» ÇM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3124289023488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61108296091648 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚64088007144960 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3124201586688 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59448896585792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3124201586688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚282 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61108320995328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 % _timestamp‚1.7857949076787152e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61098856 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚64088007144960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3124289023488 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61089184 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64317176221696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚62565554687098 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23593870 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3116555258710 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚60963718121472 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37504984 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64317010165248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚60963701082112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24153140 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37504984 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61098856 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59448913625152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265šîS2¼ - _wandbruntime‚1581  _runtime‚1581èE­œ½ û 0loss/FRt1/assignment_loss‚2.9620825893258393 1loss/FBt2/assignment_loss‚0.05551350324158728 0loss/SRqqt2/detection_loss‚2.101244770818304 &loss/total_loss‚26.842054470724083 trainer/global_step‚3549 % _timestamp‚1.7857949076809416e+09 1loss/SRqqt2/assignment_loss‚1.833452846441004 0loss/FBt1/assignment_loss‚0.6909382715821266 /loss/FRt2/detection_loss‚3.0365184175116675 /loss/FBt1/detection_loss‚3.3948372304439545 _runtime‚1581.757725515 /loss/FRt2/assignment_loss‚4.558262862477984 .loss/FBt2/detection_loss‚2.651229055963505 2loss/SRqqt1/assignment_loss‚1.6557675260340883 .loss/FRt1/detection_loss‚2.773424425149617 0loss/SRqqt1/detection_loss‚1.128782971734403 epoch‚0 _step‚283âúx»¾ € 0loss/FRt1/assignment_loss‚2.9620825893258393 1loss/FBt2/assignment_loss‚0.05551350324158728 0loss/SRqqt2/detection_loss‚2.101244770818304 &loss/total_loss‚26.842054470724083 trainer/global_step‚3549 % _timestamp‚1.7857949076809416e+09 1loss/SRqqt2/assignment_loss‚1.833452846441004 0loss/FBt1/assignment_loss‚0.6909382715821266 /loss/FRt2/detection_loss‚3.0365184175116675 /loss/FBt1/detection_loss‚3.3948372304439545 _runtime‚1581.757725515 /loss/FRt2/assignment_loss‚4.558262862477984 .loss/FBt2/detection_loss‚2.651229055963505 2loss/SRqqt1/assignment_loss‚1.6557675260340883 .loss/FRt1/detection_loss‚2.773424425149617 0loss/SRqqt1/detection_loss‚1.128782971734403 epoch‚0 _step‚283›6;ï¿ ‚(:… ã¢ÄÓ¦€ú¸ gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚97.206' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚32.402 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è jÀ ‚(:ß  ã¢ÄÓϦǹ disk.sda1.in‚0 disk.drbd1356.in‚0 cpu‚13.903778078222906) proc.memory.availableMB‚ 38524.4140625 disk.dm-0.in‚0.734375 network.sent‚ 3.374291e+06 disk.sda.in‚0 disk.drbd1931.out‚0* disk./.usagePercent‚31.895848099156428 disk.drbd1226.in‚0 disk.drbd1965.in‚0 disk.sda2.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1510.in‚0 disk.drbd2111.out‚0 disk.drbd1673.out‚0 disk.drbd1510.out‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.sda.out‚ 41.13671875 disk.drbd1931.in‚0# disk.nvme0n1.out‚41.86962890625 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.dm-0.out‚41.86962890625 network.recv‚264196 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.nvme0n1.in‚0.734375 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd2078.out‚0$ proc.memory.rssMB‚11253.34765625 disk.drbd1817.in‚0 disk.drbd1550.out‚0$ memory_percent‚41.21640920639038) proc.memory.percent‚17.17124581336975 disk.drbd1828.out‚0 disk.drbd2041.in‚0 disk.drbd1552.in‚0 disk.rbd0.out‚ 0.00390625 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd1938.out‚0 disk.rbd0.in‚ 0.02001953125 disk.sda2.out‚ 41.13671875 disk.drbd2078.in‚0 disk.drbd1226.out‚0 proc.cpu.threads‚29$ disk./.usageGB‚570.3301582336426 disk.drbd2111.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 0.771484375Z9[ì2Á - _wandbruntime‚1603  _runtime‚1603oSÆ À _step‚284 & lr-AdamW/pg1‚9.275409916061665e-05 & lr-AdamW/pg2‚9.275409916061665e-05 trainer/global_step‚3599 $ _timestamp‚1.785794929447187e+09 _runtime‚1603.524147404‘qû“Ëà Å & lr-AdamW/pg2‚9.275409916061665e-05 trainer/global_step‚3599 $ _timestamp‚1.785794929447187e+09 _runtime‚1603.524147404 _step‚284 & lr-AdamW/pg1‚9.275409916061665e-05œÆ ð32Ä - _wandbruntime‚1603  _runtime‚1603(…½¾'Å ¸O OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚61804872469504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23919379 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3252988599296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24487137 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚61951469718528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚60269182068832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38012793 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64972190572032 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38012793 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚64972190572032 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23919377 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65204591089152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚63428834120952 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3159566248996 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3167318102528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10311299 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3159652052120 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 % _timestamp‚1.7857949294481936e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚64972295041024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚64972295041024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3167318102528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚61804889508864 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚63428731278468 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3167405532160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61932170 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10311540 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64972190572032 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38012793 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚64972190572032 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23919377 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65204591089152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚63428834120952 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3159566248996 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3167318102528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10311299 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3159652052120 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 % _timestamp‚1.7857949294481936e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚64972295041024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚64972295041024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3167318102528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚61804889508864 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚63428731278468 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3167405532160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61932170 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10311540 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3168197722112 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14179534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚64990341430784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61822039232512 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3168285158912 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38023316 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24493963 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61949336 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61959008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚63446348924010 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 % _timestamp‚1.7857949298263042e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚61822056271872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23926020 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38032986 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61968651931648 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚64990236954624 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3168285158912 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚63446451766606 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚60285922241152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3160529525454 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 _runtime‚1603.903660207 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61949336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10314429 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14179532 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23926022 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24493718 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3168197722112 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14179534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚64990341430784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61822039232512 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3168285158912 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38023316 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24493963 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61949336 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61959008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚63446348924010 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 % _timestamp‚1.7857949298263042e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚61822056271872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23926020 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38032986 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61968651931648 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚64990236954624 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3168285158912 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚63446451766606 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚60285922241152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3160529525454 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 _runtime‚1603.903660207 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61949336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10314429 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14179532 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23926022 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24493718 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚66109953306112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38550451 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚0Ël€&121 % _timestamp‚1.7857949515094464e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24251527 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24826899 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62792322 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3298152651264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚3649 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24251527 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚61106190684832 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3211401667584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚62663227659264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3211314237952 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14372447 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24251529 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚61106173645472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64309628357976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚62811800654848 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10454452 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65874524857856 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38550451 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65874524857856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚65874629326848 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62801980 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64309731200460 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚62663227659264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3211401667584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24251529&uwãÂ'Ö ¼O O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65874524857856 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38550451 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65874524857856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚65874629326848 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62801980 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64309731200460 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚62663227659264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3211401667584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24251529 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38540795 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38540795 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚62663210619904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65874629326848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚62811825558528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚62663210619904 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3211314237952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24826640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14372445 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 _step‚289 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3203540515628 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚66110115175424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚66109953306112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38550451 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 % _timestamp‚1.7857949515094464e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24251527 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24826899 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62792322 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3298152651264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚3649 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24251527 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚61106190684832 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3211401667584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚62663227659264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3211314237952 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14372447 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24251529 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚61106173645472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64309628357976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚62811800654848 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10454452 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚65892571240448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38560988 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62809488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62829032675328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚61122930857152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 % _timestamp‚1.7857949518770988e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62829007771648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64327348846114 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62809488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10457012 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38551318 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24258170 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62819160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24258170 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3212193857536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24833394 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14376384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3204332185726 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _step‚290 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _runtime‚1625.954455151 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62819160 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24258172 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62680394422272 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 trainer/global_step‚3649 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚66128058997760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3299188184576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3204417988962 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚65892675716608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚65892571240448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38560988 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62809488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62829032675328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚61122930857152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 % _timestamp‚1.7857949518770988e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62829007771648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64327348846114 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62809488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10457012 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38551318 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24258170 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62819160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24258170 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3212193857536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24833394 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14376384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3204332185726 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _step‚290 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _runtime‚1625.954455151 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62819160 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24258172 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62680394422272 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 trainer/global_step‚3649 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚66128058997760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3299188184576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3204417988962¢R¤($2Ú - _wandbruntime‚1625  _runtime‚1625†p…Û ú epoch‚0 1loss/FBt2/assignment_loss‚0.16066887759069656 .loss/FRt1/detection_loss‚2.671442846387205 .loss/FRt2/detection_loss‚2.837593682386257 .loss/FBt1/detection_loss‚4.589868545532227 &loss/total_loss‚30.295316556026386 _runtime‚1625.955743101 0loss/FRt1/assignment_loss‚3.2621913258458526 2loss/SRqqt2/assignment_loss‚1.8953746894131536 0loss/SRqqt2/detection_loss‚2.716789715172601 trainer/global_step‚3649 1loss/SRqqt1/assignment_loss‚1.506887917850107 0loss/FBt1/assignment_loss‚0.4959148983859147 .loss/FBt2/detection_loss‚4.917579191570416 /loss/FRt2/assignment_loss‚4.126438211511682 1loss/SRqqt1/detection_loss‚1.1145666543802715 % _timestamp‚1.7857949518789506e+09 _step‚291ÏÂ¾Ü ÿ 1loss/SRqqt1/assignment_loss‚1.506887917850107 0loss/FBt1/assignment_loss‚0.4959148983859147 .loss/FBt2/detection_loss‚4.917579191570416 /loss/FRt2/assignment_loss‚4.126438211511682 1loss/SRqqt1/detection_loss‚1.1145666543802715 % _timestamp‚1.7857949518789506e+09 _step‚291 epoch‚0 1loss/FBt2/assignment_loss‚0.16066887759069656 .loss/FRt1/detection_loss‚2.671442846387205 .loss/FRt2/detection_loss‚2.837593682386257 .loss/FBt1/detection_loss‚4.589868545532227 &loss/total_loss‚30.295316556026386 _runtime‚1625.955743101 0loss/FRt1/assignment_loss‚3.2621913258458526 2loss/SRqqt2/assignment_loss‚1.8953746894131536 0loss/SRqqt2/detection_loss‚2.716789715172601 trainer/global_step‚3649£Rš¡Þ‘Ý ‚(:† £ÄÓÆÝ„¹ gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚126.633' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚42.211 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TÕYiÞ ‚(:Þ  £ÄÓñÚÚ¹# disk.nvme0n1.out‚43.13525390625 disk.drbd1552.in‚0 disk.rbd0.out‚ 0.0986328125 disk.sda2.in‚0 network.recv‚270088* disk./.usagePercent‚31.895896526846297 disk.rbd1.out‚ 0.771484375 disk.drbd1289.in‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd2078.in‚0 network.sent‚ 3.467474e+06 disk.drbd1285.in‚0 disk.drbd2118.in‚0 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.drbd1510.in‚0 disk.sda2.out‚ 42.296875 disk.drbd2111.out‚0 disk.drbd2041.out‚0 disk.drbd1931.in‚0$ disk./.usageGB‚570.3310241699219 disk.drbd2111.in‚0 disk.drbd1938.in‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.sda1.in‚0 cpu‚13.89978396611536 disk.drbd1226.out‚0 disk.drbd1510.out‚0) proc.memory.percent‚17.17124581336975 disk.drbd1550.out‚0% memory_percent‚41.216570138931274* proc.memory.availableMB‚38524.30859375 disk.drbd1938.out‚0 disk.nvme0n1.in‚0.734375 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1289.out‚0 disk.sda.in‚0 disk.sda.out‚ 42.296875 disk.dm-0.out‚43.13525390625 disk.drbd1285.out‚0 disk.drbd1356.out‚0 disk.drbd2078.out‚0$ proc.memory.rssMB‚11253.34765625 proc.cpu.threads‚29 disk.drbd1828.out‚0 disk.dm-0.in‚0.734375 disk.drbd1552.out‚0 disk.rbd0.in‚ 0.02001953125 disk.drbd2041.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1817.in‚0.ˆ$2ß - _wandbruntime‚1647  _runtime‚1647-– \Çà Á trainer/global_step‚3699 % _timestamp‚1.7857949736662445e+09 _runtime‚1647.743222591 _step‚292 & lr-AdamW/pg1‚9.533131780914726e-05 & lr-AdamW/pg2‚9.533131780914726e-05TŽÒ­Ìá Æ & lr-AdamW/pg2‚9.533131780914726e-05 trainer/global_step‚3699 % _timestamp‚1.7857949736662445e+09 _runtime‚1647.743222591 _step‚292 & lr-AdamW/pg1‚9.533131780914726e-05¤jÙs2â - _wandbruntime‚1647  _runtime‚1647Ñš¨r¾'ã ¸O QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚65190525437484 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7857949736673486e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3343516547072 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚65190628279968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14569297 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚66776963612672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚61943199300832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24583679 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39068797 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66776859143680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63662132 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14569295 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10598009 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10598261 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚63521548770304 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 trainer/global_step‚3699 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚63521565809664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3255310373376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39078453 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39068797 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63652474 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚63521565809664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚63521548770304 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63672156494848 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3247428979136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _runtime‚1647.744635597 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚66776963612672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67015538173440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63672181398528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144MeŠöÃ'ä ½O LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 trainer/global_step‚3699 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚63521565809664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3255310373376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39078453 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39068797 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63652474 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚63521565809664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚63521548770304 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63672156494848 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3247428979136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _runtime‚1647.744635597 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚66776963612672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67015538173440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63672181398528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚65190525437484 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7857949736673486e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3343516547072 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚65190628279968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14569297 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚66776963612672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚61943199300832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24583679 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39068797 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66776859143680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63662132 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14569295 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10598009 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10598261 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚63521548770304 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2¥!²“…2å - _wandbruntime‚1648  _runtime‚1648V“ñæ ÁM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3256189992960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10600840 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3248306452470 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66795010002432 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63679312 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63689363611648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 trainer/global_step‚3699 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚66794905526272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25174072 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚61959922433792 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚66795010002432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24590320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3344287364096 E=DeviceStatsMonitor.on_train_batch_en2-cOÖd/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39088990 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 _runtime‚1648.111818784 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63679312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚61959939473152 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚294 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24590322 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67033806546432 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67033650975744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3344418031104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚63538732572672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3256277429760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10601068 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3256277429760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚63538715533312 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚63538732572672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3248220649234 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251%Ãg-Ì&ç ÆM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚65208143083026 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚65208245925622 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚66795010002432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24590320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3344287364096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39088990 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 _runtime‚1648.111818784 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63679312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚61959939473152 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚294 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24590322 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67033806546432 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67033650975744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3344418031104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚63538732572672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3256277429760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10601068 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3256277429760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚63538715533312 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚63538732572672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3248220649234 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3256189992960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10600840 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3248306452470 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66795010002432 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63679312 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63689363611648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 trainer/global_step‚3699 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚66794905526272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25174072 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚61959922433792 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67679193429504 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3299306508800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14766145 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25507569 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64512626 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3299393938432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3388586342912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 _step‚297 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24915829 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25507313 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39606455 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚67679193429504 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚62780207916832 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24915829 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64379886920704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3388723308544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 $ _timestamp‚1.785794995940902e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3291317442644 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3291231639520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3299393938432 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67921098677760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39606455 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚1670.018061774¶"[šòô ¼O E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64512626 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor(E@AÐ&.on_train_batch_start/active_bytes.all.current‚ 104468992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3299393938432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3388586342912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 _step‚297 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24915829 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25507313 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39606455 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚67679193429504 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚62780207916832 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24915829 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64379886920704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3388723308544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 $ _timestamp‚1.785794995940902e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3291317442644 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3291231639520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3299393938432 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67921098677760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39606455 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚1670.018061774 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚67679297898496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39596799 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚64379903960064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10741168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚66071422516992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14766147 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67679193429504 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3299306508800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14766145 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25507569 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0©üáÞ2õ - _wandbruntime‚1670  _runtime‚1670_:¢ƒÇ&ö ÁM _runtime‚1670.386439581 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25513973 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24922472 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3300273565184 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39607322 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.785794996309125e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3300186128384 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚67697344288256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67939211342848 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64549744355328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3389624655360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39607322 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚64397070723072 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66089040162534 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24922472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10744127 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64539464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39616992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚64397070723072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3300186128384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64529792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚62796931049792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64549719451648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39616992 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66089143005130 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3292194915978 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24922470 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25514211 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3300273565184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3292109112742 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64397053683712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24922470 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚238 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14770082 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0"w/öÌ&÷ ÆM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25514211 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3300273565184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3292109112742 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64397053683712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24922470 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚238 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14770082 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1670.386439581 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25513973 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24922472 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3300273565184 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39607322 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.785794996309125e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3300186128384 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚67697344288256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67939211342848 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64549744355328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3389624655360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39607322 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚64397070723072 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66089040162534 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24922472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10744127 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64539464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39616992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚64397070723072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3300186128384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64529792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚62796931049792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64549719451648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39616992 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66089143005130 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3292194915978 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24922470 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160ª~S 2ø - _wandbruntime‚1670  _runtime‚1670kôù þ /loss/FRt2/assignment_loss‚4.168057206131163 1loss/FBt1/assignment_loss‚0.48272410006902644 /loss/FRt2/detection_loss‚2.6329150029591153 2loss/SRqqt1/assignment_loss‚1.4637995655131009 1loss/FBt2/assignment_loss‚0.11647031662907562 1loss/SRqqt1/detection_loss‚1.2953273742071663 0loss/SRqqt2/detection_loss‚2.819139926206498 % _timestamp‚1.7857949963108313e+09 /loss/FRt1/detection_loss‚2.3702983998803684 .loss/FBt2/detection_loss‚1.865562761025672 &loss/total_loss‚26.860240819478964 epoch‚0 trainer/global_step‚3749 0loss/FRt1/assignment_loss‚3.3437122350097983 2loss/SRqqt2/assignment_loss‚1.9506363485540663 .loss/FBt1/detection_loss‚4.351597583293914 _runtime‚1670.387609008 _step‚299ñ9t ú ƒ /loss/FRt1/detection_loss‚2.3702983998803684 .loss/FBt2/detection_loss‚1.865562761025672 &loss/total_loss‚26.860240819478964 epoch‚0 trainer/global_step‚3749 0loss/FRt1/assignment_loss‚3.3437122350097983 2loss/SRqqt2/assignment_loss‚1.9506363485540663 .loss/FBt1/detection_loss‚4.351597583293914 _runtime‚1670.387609008 _step‚299 /loss/FRt2/assignment_loss‚4.168057206131163 1loss/FBt1/assignment_loss‚0.48272410006902644 /loss/FRt2/detection_loss‚2.6329150029591153 2loss/SRqqt1/assignment_loss‚1.4637995655131009 1loss/FBt2/assignment_loss‚0.11647031662907562 1loss/SRqqt1/detection_loss‚1.2953273742071663 0loss/SRqqt2/detection_loss‚2.819139926206498 % _timestamp‚1.7857949963108313e+09«ÐfÇ‘û ‚(:† ½£Äәݙ¹ gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚117.282' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.094 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼@ "\ü ‚(:â  ½£ÄÓƒä¹ network.sent‚ 3.560866e+06$ disk./.usageGB‚570.3310432434082 disk.drbd1965.out‚0 disk.rbd0.in‚ 0.02001953125 disk.drbd1289.out‚0 disk.drbd1226.in‚0* disk./.usagePercent‚31.895897593535498 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd2111.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0$ memory_percent‚41.21667146682739 disk.drbd1285.in‚0 disk.drbd1552.out‚0 network.recv‚275790 disk.nvme0n1.in‚0.734375 disk.rbd0.out‚ 0.16162109375 disk.drbd1289.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1356.out‚0 disk.drbd1584.in‚0 disk.drbd1285.out‚0 disk.rbd1.out‚ 0.771484375 disk.drbd2118.in‚0 cpu‚13.890412126066423 disk.drbd2111.out‚0) proc.memor˜^Ñ‹y.percent‚17.17124581336975# disk.nvme0n1.out‚45.51318359375 disk.drbd2041.out‚0 disk.dm-0.in‚0.734375 disk.drbd1673.in‚0 disk.sda.in‚0 disk.dm-0.out‚45.51318359375 disk.drbd1356.in‚0 disk.drbd1510.in‚0 disk.drbd1828.in‚0 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.drbd1828.out‚0) proc.memory.availableMB‚ 38524.2421875 proc.cpu.threads‚29 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1931.in‚0 disk.drbd1673.out‚0$ proc.memory.rssMB‚11253.34765625 disk.drbd2078.out‚0 disk.sda.out‚ 43.31640625 disk.drbd2041.in‚0 disk.drbd1965.in‚0 disk.sda2.out‚ 43.31640625 disk.drbd2078.in‚0 disk.drbd2118.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 ¦Ô 2ý - _wandbruntime‚1691  _runtime‚1691dŒÄËÆþ À _runtime‚1691.731400097 _step‚300 & lr-AdamW/pg1‚9.790853645767785e-05 & lr-AdamW/pg2‚9.790853645767785e-05 trainer/global_step‚3799 $ _timestamp‚1.785795017654434e+09¨˜€eËÿ Å $ _timestamp‚1.785795017654434e+09 _runtime‚1691.731400097 _step‚300 & lr-AdamW/pg1‚9.790853645767785e-05 & lr-AdamW/pg2‚9.790853645767785e-05 trainer/global_step‚3799¬£§ô2€ - _wandbruntime‚1691  _runtime‚1691é¹ÝC¼' ¶O dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚65392893078528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 # _timestamp‚1.78579501765548e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63617216532832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3335120103028 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25247979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14962995 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25247977 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚66952319596500 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65238242110464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3335205906152 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚68581527715328 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25247979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68581527715328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25846431 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3343302644224 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚65238225071104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25247977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68581632184320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚65238242110464 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚68826834272768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65382436 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3433941194240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _step‚301 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚68826670306304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14962997 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65238225071104 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚68581632184320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚1691.732735655 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538A£CÁ'‚ »O JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25247979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14962995 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25247977 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚66952319596500 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65238242110464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3335205906152 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚68581527715328 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25247979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68581527715328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25846431 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3343302644224 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚65238225071104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25247977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68581632184320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚65238242110464 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚68826834272768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65382436 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3433941194240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _step‚301 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚68826670306304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14962997 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65238225071104 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚68581632184320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚1691.732735655 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚65392893078528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 # _timestamp‚1.78579501765548e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63617216532832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3335120103028 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251­§ÌÍb2ƒ - _wandbruntime‚1692  _runtime‚1692gøD¾È&„ ÂM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25254622 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3344269700608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10885983 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25254620 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40144994 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40135324 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66969937242042 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚65255391834112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 % _timestamp‚1.7857950180356016e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65399616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66970040084638 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65389944 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚68599678574080 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68599574097920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25852915 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25853149 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚68599678574080 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3335997576250 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3799 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3344182263808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3434836616192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _step‚302 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3344182263808 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚65255408873472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25254620 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65399616 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚65255408873472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3336083379486 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚1692.112948193 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚1692.112948193 C;DeviceStatŸ ¤"sMonitor.on_train_batch_end/segment.all.allocated‚251 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65399616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66970040084638 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65389944 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚68599678574080 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68599574097920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25852915 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25853149 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚68599678574080 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3335997576250 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3799 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3344182263808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3434836616192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _step‚302 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640®;Uf½2† - _wandbruntime‚1692  _runtime‚1692 ¸\‡ ý _runtime‚1692.114277426 2loss/SRqqt1/assignment_loss‚1.6259299150213253 0loss/FBt1/assignment_loss‚0.6792524460425529 1loss/FBt2/assignment_loss‚0.06161628818606515 /loss/FBt2/detection_loss‚1.9603373142095601 trainer/global_step‚3799 _step‚303 0loss/FRt1/assignment_loss‚3.2891984581947327 /loss/FRt2/detection_loss‚2.4700994425349765 1loss/SRqqt1/detection_loss‚1.0858087663831146 1loss/SRqqt2/detection_loss‚4.5382662989199165 $loss/total_loss‚34.9873880746469 epoch‚0 $ _timestamp‚1.785795018037472e+09 1loss/SRqqt2/assignment_loss‚1.319834692031145 /loss/FRt1/detection_loss‚2.6922660834259458 0loss/FRt2/assignment_loss‚3.8893700320311266 /loss/FBt1/detection_loss‚11.375408337666439púñàˆ ‚ 0loss/FRt2/assignment_loss‚3.8893700320311266 /loss/FBt1/detection_loss‚11.375408337666439 _runtime‚1692.114277426 2loss/SRqqt1/assignment_loss‚1.6259299150213253 0loss/FBt1/assignment_loss‚0.6792524460425529 1loss/FBt2/assignment_loss‚0.06161628818606515 /loss/FBt2/detection_loss‚1.9603373142095601 trainer/global_step‚3799 _step‚303 0loss/FRt1/assignment_loss‚3.2891984581947327 /loss/FRt2/detection_loss‚2.4700994425349765 1loss/SRqqt1/detection_loss‚1.0858087663831146 1loss/SRqqt2/detection_loss‚4.5382662989199165 $loss/total_loss‚34.9873880746469 epoch‚0 $ _timestamp‚1.785795018037472e+09 1loss/SRqqt2/assignment_loss‚1.319834692031145 /loss/FRt1/detection_loss‚2.6922660834259458¯+ÚïZ‰ ‚(:… Ì£ÄÓ¶ÿô¸ gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚99.399' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚33.133 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W3ÑKdŠ ‚(:Ù  Ì£ÄÓª…¼¹ network.recv‚278144$ disk./.usageGB‚570.3310279846191 disk.nvme0n1.in‚0.734375 disk.drbd1289.in‚0 disk.sda1.out‚0 disk.drbd1938.in‚0) proc.memory.percent‚17.17124581336975 cpu‚13.88773180668162 disk.sda.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1356.out‚0 disk.sda2.out‚43.78125 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0$ memory_percent‚41.21715426445007 proc.cpu.threads‚29 disk.rbd0.out‚ 0.224609375 disk.drbd2111.in‚0 disk.drbd1584.out‚0$ proc.memory.rssMB‚11253.34765625 disk.sda2.in‚0 disk.drbd2118.in‚0 disk.drbd1965.in‚0 disk.dm-0.out‚45.97705078125 network.sent‚ 3.605819e+06 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd2041.in‚0 disk.drbd1931.out‚0 disk.rbd0.in‚ 0.02001953125 disk.drbd2041.out‚0 disk.drbd1673.in‚0) disk./.usagePercent‚31.89589674018414 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.drbd1285.in‚0 disk.drbd1965.out‚0 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0# disk.nvme0n1.out‚45.97705078125 disk.drbd1552.in‚0 disk.rbd1.out‚ 0.771484375 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.rbd1.in‚0 disk.drbd2078.out‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.sda.out‚43.78125 disk.dm-0.in‚0.734375 disk.drbd1938.out‚0* proc.memory.availableMB‚38523.92578125ò`1œ‹ ‚(:‘ Û£ÄÓ¨»¹ gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1587066650390625) gpu.0.memoryAllocatedBytes‚ 3572301824 gpu.0.temp‚38.0 gpu.0.powerWatts‚121.069' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.35633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^ßQZjŒ ‚(:ß  Û£ÄÓЄڹ network.sent‚ 3.650764e+06 disk.sda2.in‚0 disk.sda2.out‚ 44.01171875 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.rbd1.out‚ 0.771484375 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd1817.out‚0 cpu‚13.884524956659298 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0# proc.memory.rssMB‚ 11253.3515625) proc.memory.percent‚17.17125177383423 proc.cpu.threads‚29 disk.drbd1552.in‚0 disk.drbd1289.out‚0 disk.drbd1550.out‚0 disk.drbd1828.in‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd1931.out‚0 disk.dm-0.out‚46.19970703125) disk./.usagePercent‚31.89589674018414 disk.drbd1828.out‚0 disk.drbd1817.in‚0 disk.drbd1285.out‚0* proc.memory.availableMB‚38523.92578125 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1584.out‚0 network.recv‚280359 disk.drbd1356.out‚0 disk.sda.in‚0 disk.sda.out‚ 44.01171875 disk.drbd1931.in‚0 disk.dm-0.in‚0.734375 disk.drbd1938.in‚0$ disk./.usageGB‚570.3310279846191 disk.drbd2111.in‚0 disk.drbd1938.out‚0 disk.drbd1510.in‚0 disk.drbd1285.in‚0 disk.rbd0.in‚ 0.02001953125 disk.rbd0.out‚ 0.224609375 disk.sda1.in‚0 disk.drbd1226.out‚0$ memory_percent‚41.21715426445007 disk.sda1.out‚0 disk.nvme0n1.in‚0.734375# disk.nvme0n1.out‚46.199707031258ú™À2 - _wandbruntime‚1714  _runtime‚1714fZ΋ÅŽ ¿ & lr-AdamW/pg1‚9.919714578194316e-05 & lr-AdamW/pg2‚9.919714578194316e-05 trainer/global_step‚3849 $ _timestamp‚1.785795039947243e+09 _runtime‚ 1714.02426802 _step‚3044tFÊ Ä _step‚304 & lr-AdamW/pg1‚9.919714578194316e-05 & lr-AdamW/pg2‚9.919714578194316e-05 trainer/global_step‚3849 $ _timestamp‚1.785795039947243e+09 _runtime‚ 1714.02426802°ÇÒj&2 - _wandbruntime‚1714  _runtime‚1714EéWú¾'‘ ¸O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3387298779648 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚66096563221504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40652803 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25580129 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25580127 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚64454225148832 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚66096580260864 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69483862001152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 % _timestamp‚1.7857950399483879e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3387298779648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚69483966470144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26186032 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40662459 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚69483966470144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3387386209280 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3379094369660 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25580127 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15159845 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3479147313664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _runtime‚1714.025668093 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25580129 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3387386209280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚69483862001152 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚66096563221504 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚67833216676008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 _step‚305 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚69732396232192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26186277 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3479016639488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66232930 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66242588 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3379008566536cÍ>&’ ½O E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66232930 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66242588 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3379008566536 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3387298779648 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚66096563221504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40652803 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25580129 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25580127 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚64454225148832 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚66096580260864 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69483862001152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 % _timestamp‚1.7857950399483879e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3387298779648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚69483966470144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26186032 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40662459 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚69483966470144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3387386209280 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3379094369660 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25580127 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15159845 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3479147313664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _runtime‚1714.025668093 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25580129 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9Devi