:W&Bá¾’š%VM apluhd37ª; 9 /0.28.1+bc2cf24b1bd9fa3a1b6cbeb920087b832afc526f0.70.0KàÁ‚Â" rb3syqre35t0 apluhd37ŠŸ apluhd37spatop" _wandb‚{}Bspatop_v6_blocks_outputjdp-spatop-train-v6-blocks-7845cŠ ¢èÄÓ ‰Ë!š1 (H¨¸h"3.12.10*0.28.1j linux-x86_64gK–ÿcÒÝ -Linux-6.8.0-117-generic-x86_64-with-glibc2.36CPython 3.12.10 ¡èÄÓŠ­µ*-of*A/data/spatop/options_files/sweep_fixed_dp/config_gs8pex8v_v6.json*-l*/scratch/logs/*-n*spatop_v6_blocks_output2-m spanet.trainRdanielprimosch@gmail.comZ/scratch/logs/bdp-spatop-train-v6-blocks-7845cr/opt/conda/bin/pythonº lczh9ilhpdbzl4i9yrekgjfn8fcx2kgk¦kd~ÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€ ­À§&ª€ÀåÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-90988967-60ae-29dd-fe10-539d0dd12c25Ê13.2º lczh9ilhpdbzl4i9yrekgjfn8fcx2kgkæPò2  requirements.txt®Gy9P apluhd37Z? (H¨¸ (H¨¸h"3.12.10*0.28.1j linux-x86_64 #ˆüR apluhd37ZA (H¨¸ (H¨¸8h"3.12.10*0.28.1j linux-x86_64Ϥx\* apluhd37b trainer/global_step2XWøtU  apluhd37ZD (H¨¸ (H¨¸8h"3.12.10*0.28.1j linux-x86_64$fW/  apluhd37b*"trainer/global_step2qTPK  apluhd37j: ¢èÄÓ¸¼ÿŽ*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] J’”}“  apluhd37j  ¢èÄÓ˜ÄÐÃî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 220 K │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 332 K │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 4.8 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 12 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ } ý\  apluhd37jÊ ¢èÄÓÐüÂÄ·Trainable params: 5.3 M Non-trainable params: 50 Total params: 5.3 M Total estimated model params size (MB): 21.374 Modules in train mode: 2326 Modules in eval mode: 0 Total FLOPs: 0 9«í½ apluhd37*«  hidden_dim‚64  transformer_dim‚64  transformer_dim_scale‚2.0  initial_embedding_dim‚64  position_embedding_dim‚64  num_embedding_layers‚10  num_encoder_layers‚4 ! num_branch_embedding_layers‚3  num_branch_encoder_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1  num_attention_heads‚8 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 K event_info_file‚7"/data/spatop/event_files/v11/tt_hadronic_v7_full.yaml" K training_file‚9"/data/spatop/delphes_v6/tt_hadronic_training2k_clean.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_limit‚1.0  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  batch_size‚512  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW" ' learning_rate‚0.0007344299982717679  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0  dropout‚0.2978723651141102 epochs‚10 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"K|4ë apluhd37jÙ ¢èÄÓ€›þÇÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. +›ÃœN apluhd37j¼ ¢èÄÓ‹È«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=7` in the `DataLoader` to improve performance. ŒU¢ë apluhd37jÙ ¥èÄÓèµÓòÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. Út/dP apluhd37j¾ ¥èÄÓÐþçò­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=7` in the `DataLoader` to improve performance. §}®º›‚(:‘ ±èÄÓ„ŽåÞ gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚37.0 gpu.0.powerWatts‚123.266' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚68.48111111111112 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔÆaò,‚(:¢ ±èÄÓß”²ç network.sent‚42738% disk./.usageGB‚1225.8759117126465 disk.rbd2.in‚0 cpu‚15.835721292612673 disk.sda1.out‚0 disk.nvme0c0n1.out‚ 0.3984375 disk.rbd0.out‚0$ proc.memory.rssMB‚11266.04296875 disk.sda.in‚0 disk.sda.out‚ 0.75390625 disk.rbd2.out‚0$ memory_percent‚41.19766354560852 disk.rbd5.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.nvme0n1.out‚ 0.3984375* proc.memory.percent‚17.190617322921753 proc.cpu.threads‚29 disk.rbd5.in‚0 disk.dm-0.in‚0 disk.rbd4.in‚0 disk.drbd1743.in‚0 disk.drbd1743.out‚ 0.01953125 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚82.26956101343593 disk.dm-0.out‚ 0.3984375 disk.sda2.in‚0 disk.nvme0c0n1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 0.75390625* proc.memory.availableMB‚38536.69921875 network.recv‚44723 disk.rbd4.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 0.75390625 disk.nvme0n1.in‚0 disk.rbd0.in‚0%¬Ä93b/ lr-AdamW/pg1"trainer/global_step2:JP-s`3b/ lr-AdamW/pg2"trainer/global_step2:JPiþK®-) _wandbruntime‚28  _runtime‚28qߌÁ¼ ' lr-AdamW/pg1‚1.2628371377799987e-06 ' lr-AdamW/pg2‚1.2628371377799987e-06 trainer/global_step‚49 $ _timestamp‚1.785803838830635e+09 _runtime‚ 28.104473811 _step‚0‘=þ ' lr-AdamW/pg1‚1.2628371377799987e-06 ' lr-AdamW/pg2‚1.2628371377799987e-06 trainer/global_step‚49 $ _timestamp‚1.785803838830635e+09 _runtime‚ 28.104473811 _step‚0Ñ/òeba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP¦°nbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP@R…J]bY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JP³ñ‚$_b[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JP˜•Í!nbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPíØbÝnbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPí¥Ì½h bd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JPÆÍ¡_g!bc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JP@“«Åi"be BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JPc‚Ýh#bd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JPs@ q$bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPdÏè(i%be BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JP QÃj&bf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JP.»“Ôn'bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JPiÞc±f(bb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JP/W)bS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JPÎ9+Àm*bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JPD‘b^h+bd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JP#ÿ\bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPÆ6¯/p?bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPbjn@bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPôp2ÑoAbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPëÃmBbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JP>¤·oCbk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JP`3îhDbd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JP;XêgEbc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JP³bͳnFbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPé±ÜxGbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP;QËä`Hb\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JP'^ ´iIbe BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JP¿¸mJbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP…ÿBhiKbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JP›…tLbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPÌóB)_Mb[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JPÐÉE-sNbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JP»«š»kObg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JPëA…mrPbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPÅÁ œkQbg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JPqõ.M^RbZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JPÑÓÍ,qSbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JP;©“djTbf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JPéд–gUbc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JP“ª‘ÚvVbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JP>2èWtWbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP(&¿OsXbo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP§F^YbZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JPÑ…2¦pZbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP¦l¬Äp[bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPÒiÞ g\bc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JPQmüf]bb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JP¤ç*k^bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JP‚Ó çq_bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JP 5wÐi`be BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JPœªÃkabg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JP!ÂÝûfbbb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JPƒáÞUbcb^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JPé_[(adb] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JPßoo²mebi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPªî“efba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPšÀ Ëlgbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JPµßÝehba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPñ/Áæoibk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JP=X”Tgjbc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JPr£Ö|ikbe BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JPƒÅglbc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JPøÔp›smbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP@Š˜qnbm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JP´ƒëYsobo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP<¸¸\jpbf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JP!݇Ódqb` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JP Ÿ,Éerba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JP °”—psbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPQéh–mtbi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPû¬Èsubo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP,cvb_ Çj}bf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JPá¹õi~be BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JPÖ‚öqrbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPQŸm€bh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JPkþàpbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JP953`‚b[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JPpyÀhƒbc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPX›ƒ©o„bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPÇy£vs…bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP×*Wk†bf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JP(ókq‡bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JP‘ëódˆb_ DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JP9Fþµm‹bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPW$lŒbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JP̆Ügbb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP÷ô]IeŽb` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JP7À[Þibd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JPîvnémbh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JPj—Pm‘bh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JP`0è]t’bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JP—Ë_h“bc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JP|g”bb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JPÓmpU.•) _wandbruntime‚28  _runtime‚28Ÿ¹F{&–ÞN VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 GDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 43592487424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 841553293472 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940454912 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 862880830464 [JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 911253711360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 WFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 911100230656 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 43571143052 GDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 906456278528 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 862863791104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940454912 RADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 906456278528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 841570332832 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11814 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚144329 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 906560747520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250522112 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚514 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 862863791104 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚336729 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11814 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚49 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NCDeviceStatsMonitor.on_train_batch_sL “étart/active.large_pool.allocated‚336729 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚534307 % _timestamp‚1.7858038388322842e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658Õ-yŸh'—âN YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 43679917056 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚343574 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚524651 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250522112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚336727 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11885 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 43592487424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 841553293472 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940454912 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 862880830464 [JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 911253711360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 WFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 911100230656 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 43571143052 GDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 906456278528 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 862863791104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940454912 RADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 906456278528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 841570332832 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11814 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚144329 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 906560747520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250522112 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚514 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 862863791104 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚336729 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11814 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚49 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚336729 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚534307 % _timestamp‚1.7858038388322842e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 GDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPé¸RAmŸbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPQ1ÿi bd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JP¯+n¡bi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JPˆvÆa¢b\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JP ‡bh£bc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JP÷7óX`¤b[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JP}¼O´r¥bm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JP˜l¨h¦bc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JP©e˜gj§be BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JPJ¯ào¨bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JP,.fl©bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPTéOšuªbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JP£N[m«bh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JPû –c¬b^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JP™ÖÅe­b` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JPÕt›‘m®bh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JP55W£o¯bj GDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current"trainer/global_step2:JPbDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP(bRìd¼b_ n¿bi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPK¤ògsÀbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP ˳jÁbe BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JP<õ,ÑgÂbb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JPípÃbk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPØ-–ÒdÄb_ DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPof°hÌbc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JP›| wpÍbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JPF鱬jÎbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JPÚ0ß2kÏbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JP`ª¼ _ÐbZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPgʧÏoÑbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPúù.ÎeÒb` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JPA´©hgÓbb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JPΈšmÔbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPãÕ\^ÕbY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JP3š>QmÖbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPš,½e×b` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JP¦} ‹hØbc @DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak"trainer/global_step2:JP’Ié oÙbj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JP}’plÚbg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JP&ÎìÑmÛbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPgpÜbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JP×È1qÝbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JPÚ,f:eÞb` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JPuEË©nßbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JP—Ì°ýaàb\ 9DeviceStatsMonitor.on_train_batch_end/segment.all.current"trainer/global_step2:JPVë×hábc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JPkhê÷râbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPq"vÐgãbb ?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current"trainer/global_step2:JPÖ O fäba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JP¨bhwhåbc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JPÞ¹&gæbb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPÔ~Êr`çb[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JP‰©‚ykèbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPߥtrébm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JPÒÇ¡Zfêba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP–ÕU!nëbi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JPŒâêwìbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPæ‹‹ihíbc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JP ÀôÅfîba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JP´nˆdøb_ DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPʨAe€b` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JP¹µ‚gbb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JP4¾Lq‚bl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JP’˜Œzkƒbf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JP!®ˆm„bh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPbpr$l…bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JP”qb“]†bX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JPa ·'m‡bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JP“;oˆbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP /Ž×k‰bf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JP3‰>kŠbf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JPϺÖÝq‹bl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JP(­×bŒb] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JPnäobj GDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JP%ðºwfŽba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPìL|mbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JP ÙåÏqbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPØÎ.aV‘bQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JP¯Ddkb’b] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JP_ØžT.“) _wandbruntime‚28  _runtime‚28!ßGQ!”æL E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚888216 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 44559543808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚147431 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 TCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 924607137280 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚343370 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 44448616386 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 44472107008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 858310505152 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 880047593472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122278400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚343370 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚343372 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚147176 UDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 929216899584 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 883412195328 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚544844 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 883387291648 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 M6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 880030554112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 _runtime‚ 28.48696832 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚535174 P?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 902656278942 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚203182 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 trainer/global_step‚49 _step‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚203184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 $ _timestamp‚1.785803839213171e+09 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 44362813150 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 Q@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 924607137280 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 P?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 924502661120 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 44472107008 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 44559543808 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚888216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147182080 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 45951886336 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚878544 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚343372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚350358 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚514 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚544844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 TCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 902759121538 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640¯êŸ¬p&•êL J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 TCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 924607137280 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚343370 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 44448616386 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 44472107008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 858310505152 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 880047593472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122278400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚343370 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚343372 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚147176 UDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 929216899584 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 883412195328 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚544844 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 883387291648 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 M6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 880030554112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 _runtime‚ 28.48696832 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚535174 P?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 902656278942 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚203182 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 trainer/global_step‚49 _step‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚203184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 $ _timestamp‚1.785803839213171e+09 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 44362813150 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 Q@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 924607137280 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 P?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 924502661120 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 44472107008 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 44559543808 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚888216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147182080 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 45951886336 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚878544 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚343372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚350358 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚514 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚544844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 TCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 902759121538 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚888216 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 44559543808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚147431 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2»çëA–b< loss/FRt2/assignment_loss"trainer/global_step2:JPéÝŠA—b< loss/FBt1/assignment_loss"trainer/global_step2:JP2ñ@˜b; loss/FBt1/detection_loss"trainer/global_step2:JP®š#A™b< loss/FRt1/assignment_loss"trainer/global_step2:JPN;7Cšb> loss/SRqqt2/assignment_loss"trainer/global_step2:JP­©Ði@›b; loss/FRt1/detection_loss"trainer/global_step2:JPL™Bœb= loss/SRqqt2/detection_loss"trainer/global_step2:JP«§g@b; loss/FBt2/detection_loss"trainer/global_step2:JPÌ<&Ç7žb2 loss/total_loss"trainer/global_step2:JPÕÏ¿W-Ÿb( epoch"trainer/global_step2:JPƒcòC b> loss/SRqqt1/assignment_loss"trainer/global_step2:JPK•…w@¡b; loss/FRt2/detection_loss"trainer/global_step2:JP/‹ÞB¢b= loss/SRqqt1/detection_loss"trainer/global_step2:JPJW.fA£b< loss/FBt2/assignment_loss"trainer/global_step2:JPëèËÂ.¤) _wandbruntime‚28  _runtime‚28ë¨Fxó¥í .loss/FRt1/detection_loss‚4.559398133111627 1loss/SRqqt2/detection_loss‚3.7218787314680717 .loss/FBt2/detection_loss‚21.92648441427284 %loss/total_loss‚76.36036418031617 epoch‚0 % _timestamp‚1.7858038392150016e+09 1loss/SRqqt1/assignment_loss‚4.471143444764961 .loss/FRt2/detection_loss‚4.084187614114097 0loss/SRqqt1/detection_loss‚3.595424144881443 trainer/global_step‚49 _runtime‚ 28.488108568 0loss/FBt2/assignment_loss‚2.3863872289657593 .loss/FRt2/assignment_loss‚5.83957552106193 .loss/FBt1/assignment_loss‚2.37303835612077 .loss/FBt1/detection_loss‚13.15067571745469 _step‚3 /loss/FRt1/assignment_loss‚5.671232907395614 1loss/SRqqt2/assignment_loss‚4.580937966704369’È3U÷¦ñ 1loss/SRqqt2/detection_loss‚3.7218787314680717 .loss/FBt2/detection_loss‚21.92648441427284 %loss/total_loss‚76.36036418031617 epoch‚0 % _timestamp‚1.7858038392150016e+09 1loss/SRqqt1/assignment_loss‚4.471143444764961 .loss/FRt2/detection_loss‚4.084187614114097 0loss/SRqqt1/detection_loss‚3.595424144881443 trainer/global_step‚49 _runtime‚ 28.488108568 0loss/FBt2/assignment_loss‚2.3863872289657593 .loss/FRt2/assignment_loss‚5.83957552106193 .loss/FBt1/assignment_loss‚2.37303835612077 .loss/FBt1/detection_loss‚13.15067571745469 _step‚3 /loss/FRt1/assignment_loss‚5.671232907395614 1loss/SRqqt2/assignment_loss‚4.580937966704369 .loss/FRt1/detection_loss‚4.559398133111627·Z6Êœ§‚(:‘ ÀèÄÓ⚇Ü gpu.0.gpu‚42.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚38.0 gpu.0.powerWatts‚120.113' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚66.72944444444444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úzø<¨‚(:± ÀèÄÓ·öÛÞ) disk./.usagePercent‚82.26956613358661 disk.rbd4.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 1.4765625 disk.rbd3.out‚0 disk.rbd1.out‚0 network.recv‚46043 disk.sda.in‚0 disk.sda.out‚ 1.4765625" disk.nvme0n1.out‚ 4.02001953125 disk.sda3.in‚0 proc.cpu.threads‚29% disk./.usageGB‚1225.8759880065918# memory_percent‚41.2007212638855 disk.rbd1.in‚0 network.sent‚49478 disk.dm-1.in‚0 disk.drbd1743.out‚ 0.02734375 disk.sda1.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 11266.078125 disk.dm-0.in‚0 disk.rbd5.in‚0 disk.drbd1743.in‚0 disk.rbd2.out‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0) proc.memory.percent‚17.19067096710205 disk.rbd5.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 1.4765625) proc.memory.availableMB‚ 38534.6953125 cpu‚15.61388111266191 disk.dm-0.out‚ 4.02001953125 disk.rbd4.out‚ 0.06298828125 disk.rbd2.in‚0$ disk.nvme0c0n1.out‚ 4.02001953125¶E›©‚(: ÏèÄÓïÙÛ gpu.0.gpu‚23.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚151.55' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚84.19444444444444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óCíl`ª‚(:Õ ÏèÄÓÑ›´Þ) disk./.usagePercent‚82.34584153035324 disk.dm-0.in‚ 63.671875 disk.sda.in‚0 disk.rbd5.out‚0 disk.dm-1.out‚ 2.19921875 disk.sda1.in‚0 disk.rbd3.out‚0% memory_percent‚41.200846433639526 disk.nvme0n1.in‚ 63.671875 disk.sda3.out‚ 2.19921875 disk.drbd1743.in‚0 disk.sda1.out‚0 cpu‚15.453256339346867 disk.nvme0c0n1.in‚ 63.671875 disk.rbd5.in‚0 disk.sda2.in‚0 disk.rbd1.out‚0 disk.dm-0.out‚85.37158203125 disk.sda.out‚ 2.19921875 proc.cpu.threads‚29 network.recv‚48019% disk./.usageGB‚1227.0125465393066# disk.nvme0n1.out‚85.37158203125 disk.sda3.in‚0 disk.rbd4.out‚ 0.06298828125 disk.rbd3.in‚0) proc.memory.percent‚17.19067692756653 disk.rbd0.out‚0 disk.drbd1743.out‚ 0.03515625 disk.rbd2.out‚0% disk.nvme0c0n1.out‚85.37158203125 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda2.out‚0 network.sent‚93990 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd2.in‚0* proc.memory.availableMB‚38534.61328125$ proc.memory.rssMB‚11266.08203125¿À«É.«) _wandbruntime‚48  _runtime‚482‰a ì½ ' lr-AdamW/pg1‚2.5514464620453036e-06 ' lr-AdamW/pg2‚2.5514464620453036e-06 trainer/global_step‚99 % _timestamp‚1.7858038592093058e+09 _runtime‚ 48.484609279 _step‚4Ž÷ª¬Ç­Á ' lr-AdamW/pg1‚2.5514464620453036e-06 ' lr-AdamW/pg2‚2.5514464620453036e-06 trainer/global_step‚99 % _timestamp‚1.7858038592093058e+09 _runtime‚ 48.484609279 _step‚4Cö‰.®) _wandbruntime‚48  _runtime‚48¨\”ó|'¯öN WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 1808895033344 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 1816652223488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 1678561909472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚287303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 1721218980864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 1721201941504 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚396095 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 1726560918528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _runtime‚ 48.484710749 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚668877 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 1766038555392 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 87373803436 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 87588622848 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 87676052480 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7858038592102103e+09 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1731188 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚396097 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 1721201941504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 87459606560 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚668877 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚668879 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚287046 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 1816811995648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚522 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 1808790564352 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 1726536014848 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 90116208640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚683400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 1721218980864 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚683141 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 1678578948832 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 1808895033344 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 90251077120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 1808790564352:5àjë°úN ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _runtime‚ 48.484710749 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚668877 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 1766038555392 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 87373803436 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 87588622848 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 87676052480 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7858038592102103e+09 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1731188 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚396097 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 1721201941504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 87459606560 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚668877 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚668879 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚287046 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 1816811995648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚522 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 1808790564352 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 1726536014848 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 90116208640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚683400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 1721218980864 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚683141 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 1678578948832 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚13½€×Y•0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 1808895033344 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 90251077120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 1808790564352 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 1808895033344 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 1816652223488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 1678561909472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚287303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 1721218980864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 1721201941504 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚396095 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 1726560918528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136dÜp.±) _wandbruntime‚48  _runtime‚48zµ(†&²€M % _timestamp‚1.7858038595476654e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚675520 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 1738385743872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚675522 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1748368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 88555679232 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚675520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1738696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1695302081792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1063176 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 91018568192 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 1783553358450 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 88251276658 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 88468242432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 1834919367680 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚512 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 1738368704512 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚675522 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 88468242432 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 1826836946944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 1743768035328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 1738368704512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1834761699840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 1695319121152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 trainer/global_step‚99 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 88555679232 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1738696 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚690200 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚290166 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1748368 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1072846 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 _runtime‚ 48.821264276 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚400034 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚289925 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚689957 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚400032 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 91151332352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚675520 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 1738385743872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚675522 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1748368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 88555679232 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚675520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1738696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1695302081792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1063176 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 91018568192 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 1783553358450 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 88251276658 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 88468242432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 1834919367680 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚512 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 1738368704512 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚675522 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 88468242432 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 1826836946944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 1743768035328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 1738368704512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1834761699840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 1695319121152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 trainer/global_step‚99 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 88555679232 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1738696 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚690200 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚290166 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1748368 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1072846 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 _runtime‚ 48.821264276 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚400034 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚289925 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚689957 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚400032 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 91151332352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 131348070068 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 $ _timestamp‚1.785803881297274e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 2579540091904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 135338538496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 LDDeviceStatsMonitor.on_train_batch_­FJÄstart/segment.large_pool.allocated‚130 trainer/global_step‚149 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2581682 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1023067 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 2711124850176 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚592945 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚430122 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 2711229319168 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2711124850176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1001027 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 2579557131264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 2515570525472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1590311 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 131672187904 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 131262266944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _runtime‚ 70.57096911 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚522 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1001027 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1001029 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 2579557131264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 2586916758528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 2586891854848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 131584758272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 2579540091904 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1590311 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚592947 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚430385 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0£î‹Ê'¾‡O QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 2722388068352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 2515587564832 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1001029 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 2646935634900 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 2646832792416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 131348070068 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 $ _timestamp‚1.785803881297274e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 2579540091904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 135338538496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 trainer/global_step‚149 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2581682 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1023067 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 2711124850176 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚592945 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚430122 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 2711229319168 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2711124850176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1001027 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 2579557131264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 2515570525472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1590311 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 131672187904 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 131262266944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _runtime‚ 70.57096911 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚522 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1001027 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1001029 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 2579557131264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 2586916758528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 2586891854848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 131584758272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 2579540091904 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1590311 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚592947 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚430385 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚512 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2591340 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚9 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 131672187904 V;]‚.¿) _wandbruntime‚70  _runtime‚701;ee–&ÀM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2608520 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚433005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1030145 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 132551814656 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2598848 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 2729171232768 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2608520 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1007672 N6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 trainer/global_step‚149 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 132139740166 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1007670 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2596706854912 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚596882 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1591178 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 132464377856 % _timestamp‚1.7858038816780515e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 2532310697792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 136245420032 _step‚10 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1029887 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 2740495768064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2729275708928 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1600848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 2532327737152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1007672 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 136371892736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 70.951823047 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1007670 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚433261 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 2604098971648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 2664450437958 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚522 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 2664553280554 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 2596723894272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚596884 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1600848 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2729275708928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚512[$TñÈÁ”M EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1029887 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 2740495768064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2729275708928 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1600848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 2532327737152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1007672 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 136371892736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 70.951823047 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1007670 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚433261 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 2604098971648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 2664450437958 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚522 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 2664553280554 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 2596723894272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚596884 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1600848 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2729275708928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚512 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2608520 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚433005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1030145 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 132551814656 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2598848 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 2729171232768 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2608520 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1007672 N6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 trainer/global_step‚149 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 132139740166 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1007670 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2596706854912 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚596882 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1591178 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 132464377856 % _timestamp‚1.7858038816780515e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 2532310697792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 136245420032 _step‚10 ¤s.Â) _wandbruntime‚70  _runtime‚70ľøÃò /loss/FRt1/assignment_loss‚5.748558434573087 /loss/FRt2/assignment_loss‚5.756672798902139 epoch‚0 _step‚11 /loss/FBt2/assignment_loss‚2.304508481706892 .loss/FRt1/detection_loss‚4.561357854635684 .loss/FRt2/detection_loss‚4.090310440837652 .loss/FBt1/detection_loss‚17.82266831633292 %loss/total_loss‚77.27169712404515 _runtime‚ 70.952975938 1loss/SRqqt2/assignment_loss‚4.493526137701355 /loss/FBt1/assignment_loss‚2.318110102101376 1loss/SRqqt1/detection_loss‚3.5008663614006603 1loss/SRqqt2/detection_loss‚3.4347879700731525 /loss/FBt2/detection_loss‚18.745130893729982 % _timestamp‚1.7858038816796372e+09 1loss/SRqqt1/assignment_loss‚4.495199332050249 trainer/global_step‚149“fãßüÄö /loss/FRt1/assignment_loss‚5.748558434573087 /loss/FRt2/assignment_loss‚5.756672798902139 epoch‚0 _step‚11 /loss/FBt2/assignment_loss‚2.304508481706892 .loss/FRt1/detection_loss‚4.561357854635684 .loss/FRt2/detection_loss‚4.090310440837652 .loss/FBt1/detection_loss‚17.82266831633292 %loss/total_loss‚77.27169712404515 _runtime‚ 70.952975938 1loss/SRqqt2/assignment_loss‚4.493526137701355 /loss/FBt1/assignment_loss‚2.318110102101376 1loss/SRqqt1/detection_loss‚3.5008663614006603 1loss/SRqqt2/detection_loss‚3.4347879700731525 /loss/FBt2/detection_loss‚18.745130893729982 % _timestamp‚1.7858038816796372e+09 1loss/SRqqt1/assignment_loss‚4.495199332050249 trainer/global_step‚149 ž~¥›Å‚(: íèÄÓ‚ÏæÛ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚147.947' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚82.19277777777778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0veægÆ‚(:Ü íèÄÓïì°Þ! disk.nvme0c0n1.in‚ 88.33984375 disk.rbd1.out‚0 disk.rbd0.in‚0( proc.memory.percent‚17.1907901763916 cpu‚15.207035646367231 disk.dm-1.out‚ 3.53515625 network.recv‚52973 disk.nvme0n1.in‚ 88.33984375% disk.nvme0n1.out‚1223.64892578125 proc.cpu.threads‚29 disk.rbd4.out‚ 0.06298828125 disk.rbd1.in‚0 disk.sda3.out‚ 3.53515625 disk.rbd0.out‚0 disk.drbd1743.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚1223.64892578125 disk.dm-0.in‚ 88.33984375 disk.rbd4.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0# memory_percent‚41.2014901638031 disk.sda3.in‚0* proc.memory.availableMB‚38534.19140625" disk.dm-0.out‚1223.64892578125 disk.sda.out‚ 3.53515625 disk.drbd1743.out‚ 0.08203125 disk.sda1.in‚0 disk.dm-1.in‚0 network.sent‚141534) disk./.usagePercent‚82.34958948065452% disk./.usageGB‚1227.0683937072754 disk.sda.in‚0 disk.rbd5.in‚0 disk.rbd5.out‚0! proc.memory.rssMB‚ 11266.15625 disk.rbd2.out‚0 disk.sda2.in‚0ˆžæ‘Ç‚(:† üèÄÓá´ºÛ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚119.619' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚66.455 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÚeÈ‚(:Ú üèÄÓèÃþÝ disk.sda2.out‚0! proc.memory.rssMB‚ 11266.15625( proc.memory.percent‚17.1907901763916 disk.rbd3.in‚0' disk.nvme0c0n1.out‚1231.25732421875 disk.rbd1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 91.7265625 disk.sda1.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚1231.25732421875 disk.nvme0n1.in‚ 91.7265625 disk.sda.in‚0 disk.sda.out‚ 4.33203125 disk.rbd4.out‚ 0.06298828125$ memory_percent‚41.20174050331116 cpu‚15.116393987808756 disk.drbd1743.in‚0 network.recv‚55386) disk./.usagePercent‚82.34976023767985 disk.rbd1.in‚0 disk.rbd4.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 4.33203125 disk.drbd1743.out‚ 0.08984375 network.sent‚186357 disk.nvme0c0n1.in‚ 91.7265625 disk.sda3.out‚ 4.33203125 disk.rbd5.in‚0 disk.rbd2.in‚0 disk.sda2.in‚0* proc.memory.availableMB‚38534.02734375" disk.dm-0.out‚1231.25732421875 disk.rbd2.out‚0 proc.cpu.threads‚29% disk./.usageGB‚1227.0709381103516 disk.sda3.in‚0 disk.rbd5.out‚0 disk.sda1.out‚0ùÂ.É) _wandbruntime‚91  _runtime‚91N+ÌÿÃʽ & lr-AdamW/pg1‚5.128665110575913e-06 & lr-AdamW/pg2‚5.128665110575913e-06 trainer/global_step‚199 % _timestamp‚1.7858039017364001e+09 _runtime‚ 91.009682625 _step‚12×@£ÑÇËÁ _runtime‚ 91.009682625 _step‚12 & lr-AdamW/pg1‚5.128665110575913e-06 & lr-AdamW/pg2‚5.128665110575913e-06 trainer/global_step‚199 % _timestamp‚1.7858039017364001e+09 FZ‚.Ì) _wandbruntime‚91  _runtime‚91‰+Œ'͆O K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1333177 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 180617964544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _runtime‚ 91.014579153 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3451492 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3441834 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2108657 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚789797 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚526 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚573582 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2118313 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3627730790912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7858039017373295e+09 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3627890563072 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3527832714408 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 3437878242304 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3447272598528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 175580893696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1363379 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 3613459136000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3437878242304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3352596180832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2118313 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 175668323328 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 3613563604992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3447247694848 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3437895281664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 trainer/global_step‚199 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3613459136000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚516 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 180483096064 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 3437895281664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2108657 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 175668323328 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚573322 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 _step‚13 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚526 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚573582 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2118313 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3627730790912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7858039017373295e+09 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3627890563072 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3527832714408 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 3437878242304 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3447272598528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 175580893696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1363379 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 3613459136000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3437878242304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3352596180832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2118313 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 175668323328 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 3613563604992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3447247694848 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3437895281664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 trainer/global_step‚199 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3613459136000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚516 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 180483096064 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 3437895281664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2108657 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 175668323328 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚573322 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 _step‚13 HDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1339822 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 176114006910 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1369888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 176547950080 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 176460513280 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 3455062044672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3468672 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1339822 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 3545347517466 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1339820 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1370131 KEDeviceStatsMonitor.on_train_batch_end/reserƒ»Ó"ved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 181391134720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3369319313792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 trainer/global_step‚199 _runtime‚ 91.348102167 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3455062044672 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 3631609994752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2119180 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 176028203674 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3464454811648 _step‚14 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3455045005312 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 176547950080 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 176460513280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚526 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 3631505518592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3464479715328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3468672 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚576397 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7858039020745132e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ª Àš&Ñ”M MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3455045005312 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 176547950080 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 176460513280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚526 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 3631505518592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3464479715328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3468672 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚576397 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7858039020745132e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2119180 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1339820 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 3455045005312 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2128850 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 3545450360062 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1339822 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 176114006910 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1369888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 176547950080 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 176460513280 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 3455062044672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3468672 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1339822 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 3545347517466 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1339820 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1370131 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 181391134720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3369319313792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 trainer/global_step‚199 _runtime‚ 91.348102167 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3455062044672 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 3631609994752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2119180 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 176028203674 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3464454811648 _step‚14 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160…ŸæØ.Ò) _wandbruntime‚91  _runtime‚91Öä…ùÓó 1loss/SRqqt1/detection_loss‚3.7662612814852534 0loss/SRqqt2/detection_loss‚4.127338790031801 epoch‚0 trainer/global_step‚199 % _timestamp‚1.7858039020757856e+09 /loss/FRt1/assignment_loss‚5.671701847560822 /loss/FRt2/assignment_loss‚5.741081901959011 .loss/FRt1/detection_loss‚5.448764415960463 1loss/SRqqt1/assignment_loss‚4.525435518711172 1loss/SRqqt2/assignment_loss‚4.532287419560444 0loss/FBt1/assignment_loss‚2.3148215658524456 .loss/FRt2/detection_loss‚4.293441906571388 /loss/FBt1/detection_loss‚19.861939779099295 _runtime‚ 91.348922154 _step‚15 0loss/FBt2/assignment_loss‚2.3741326649983723 /loss/FBt2/detection_loss‚26.129581212004027 $loss/total_loss‚88.7867883037945­G´’ýÔ÷ 0loss/SRqqt2/detection_loss‚4.127338790031801 epoch‚0 trainer/global_step‚199 % _timestamp‚1.7858039020757856e+09 /loss/FRt1/assignment_loss‚5.671701847560822 /loss/FRt2/assignment_loss‚5.741081901959011 .loss/FRt1/detection_loss‚5.448764415960463 1loss/SRqqt1/assignment_loss‚4.525435518711172 1loss/SRqqt2/assignment_loss‚4.532287419560444 0loss/FBt1/assignment_loss‚2.3148215658524456 .loss/FRt2/detection_loss‚4.293441906571388 /loss/FBt1/detection_loss‚19.861939779099295 _runtime‚ 91.348922154 _step‚15 0loss/FBt2/assignment_loss‚2.3741326649983723 /loss/FBt2/detection_loss‚26.129581212004027 $loss/total_loss‚88.7867883037945 1loss/SRqqt1/detection_loss‚3.7662612814852534¥´ƒ¬œÕ‚(:‘ ‹éÄÓûæ­Û gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚38.0 gpu.0.powerWatts‚117.825' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚65.45833333333333 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NúðCbÖ‚(:× ‹éÄÓ¢š“à disk.dm-1.out‚ 5.23046875 disk.drbd1743.out‚0.09375 disk.rbd1.in‚0 disk.rbd4.out‚ 0.06298828125 disk.rbd3.in‚0* proc.memory.availableMB‚38533.74609375( proc.memory.percent‚17.1907901763916) disk./.usagePercent‚82.38512537446734 disk.rbd4.in‚0 disk.rbd5.in‚0 disk.sda1.in‚0$ memory_percent‚41.20216965675354% disk.nvme0n1.out‚1772.12451171875 disk.sda3.in‚0 disk.drbd1743.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 network.recv‚56817 disk.nvme0c0n1.in‚ 91.7265625 disk.rbd1.out‚0 disk.nvme0n1.in‚ 91.7265625 disk.rbd0.out‚0 disk.rbd2.out‚0 proc.cpu.threads‚29% disk./.usageGB‚1227.5979042053223" disk.dm-0.out‚1772.12451171875 disk.sda1.out‚0! proc.memory.rssMB‚ 11266.15625 cpu‚15.036581922332765 disk.rbd5.out‚0 disk.rbd2.in‚0 network.sent‚189007' disk.nvme0c0n1.out‚1772.12451171875 disk.dm-0.in‚ 91.7265625 disk.sda.out‚ 5.23046875 disk.sda3.out‚ 5.23046875 disk.rbd0.in‚0 disk.sda.in‚0 disk.dm-1.in‚0|Éš0×+ _wandbruntime‚113  _runtime‚113K¤ÐüÃؽ & lr-AdamW/pg1‚6.417274434841219e-06 & lr-AdamW/pg2‚6.417274434841219e-06 trainer/global_step‚249 $ _timestamp‚1.785803923922523e+09 _runtime‚ 113.195792682 _step‚16%D¶§ÇÙÁ & lr-AdamW/pg1‚6.417274434841219e-06 & lr-AdamW/pg2‚6.417274434841219e-06 trainer/global_step‚249 $ _timestamp‚1.785803923922523e+09 _runtime‚ 113.195792682 _step‚16f¾Áp0Ú+ _wandbruntime‚113  _runtime‚113ÛÐÑŒ'Û†O D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4301986 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 4296216392704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 219577029120 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2636659 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚986647 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 4408626951432 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1665329 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚527 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4307603534848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 4408729793916 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 4189587757472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4311644 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1665327 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4515793421824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 225632285184 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚716623 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 219124997084 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4533235820032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 219039193960 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 4296233432064 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚986645 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 4515793421824 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4307628438528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 219664458752 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2646315 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 4296233432064 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1703510 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1703268 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 225762959360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1665327 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚517 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 4296216392704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 4189604796832 _step‚17 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 4515897890816 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚716863 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 $ _timestamp‚1.785803923923421e+09 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4515793421824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 225632285184 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚716623 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 219124997084 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4533235820032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 Y=kŸ?HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 219039193960 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 4296233432064 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚986645 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 4515793421824 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4307628438528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 219664458752 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2646315 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 4296233432064 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1703510 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1703268 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 225762959360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1665327 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚517 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 4296216392704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 4189604796832 _step‚17 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 4515897890816 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚716863 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 $ _timestamp‚1.785803923923421e+09 H6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 _runtime‚ 113.535852978 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 4313400195072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 4313383155712 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2647182 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4319152 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1671970 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚990584 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 226665532928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 220456648704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 4426244596974 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4324810651648 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 4426347439570 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚517 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 4206344969152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7858039242622437e+09 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 220544085504 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚719465 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4324835555328 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1671970 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1671972 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 4313400195072 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4328824 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 220002470418 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4533839804416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 4533944280576 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2647182 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 219916667182 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 220456648704 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚990582 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚719708 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚527 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 trainer/global_step‚249 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 4313383155712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1710047 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1710292 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2656852 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4319152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 _step‚18 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 220544085504 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0Ýyà›&ß•M @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1671972 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4551501088256 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4551341323264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 _runtime‚ 113.535852978 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 4313400195072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 4313383155712 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2647182 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4319152 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1671970 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚990584 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 226665532928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 220456648704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 4426244596974 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4324810651648 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 4426347439570 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚517 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 4206344969152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7858039242622437e+09 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 220544085504 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚719465 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4324835555328 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1671970 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1671972 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 4313400195072 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4328824 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 220002470418 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4533839804416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 4533944280576 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2647182 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 219916667182 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 220456648704 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚990582 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚719708 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚527 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 trainer/global_step‚249 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 4313383155712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1710047 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1710292 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2656852 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4319152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 _step‚18 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 220544085504 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚189w<0à+ _wandbruntime‚113  _runtime‚113Þâ×Gøáò 0loss/SRqqt2/assignment_loss‚4.53691742212876 /loss/FRt1/detection_loss‚4.1826466954335935 /loss/FBt1/detection_loss‚16.407057631760836 %loss/total_loss‚77.93030030970007 trainer/global_step‚249 % _timestamp‚1.7858039242635264e+09 1loss/SRqqt1/assignment_loss‚4.518707114999945 .loss/FBt2/detection_loss‚21.27866021792094 _runtime‚ 113.536678506 /loss/FRt1/assignment_loss‚5.630107338835553 0loss/FRt2/assignment_loss‚5.8114342525087554 /loss/FBt1/assignment_loss‚2.332323122024536 .loss/FRt2/detection_loss‚4.107105424520613 0loss/SRqqt2/detection_loss‚3.636617936193943 epoch‚0 /loss/FBt2/assignment_loss‚2.362886150677999 0loss/SRqqt1/detection_loss‚3.125837002694607 _step‚19)j6üâö 0loss/SRqqt2/assignment_loss‚4.53691742212876 /loss/FRt1/detection_loss‚4.1826466954335935 /loss/FBt1/detection_loss‚16.407057631760836 %loss/total_loss‚77.93030030970007 trainer/global_step‚249 % _timestamp‚1.7858039242635264e+09 1loss/SRqqt1/assignment_loss‚4.518707114999945 .loss/FBt2/detection_loss‚21.27866021792094 _runtime‚ 113.536678506 /loss/FRt1/assignment_loss‚5.630107338835553 0loss/FRt2/assignment_loss‚5.8114342525087554 /loss/FBt1/assignment_loss‚2.332323122024536 .loss/FRt2/detection_loss‚4.107105424520613 0loss/SRqqt2/detection_loss‚3.636617936193943 epoch‚0 /loss/FBt2/assignment_loss‚2.362886150677999 0loss/SRqqt1/detection_loss‚3.125837002694607 _step‚19#6¾?œã‚(:‘ šéÄÓÛÇËÞ gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚150.909' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚83.83833333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„(J“Sä‚(:È šéÄÓð³ýß disk.rbd4.in‚0 disk.rbd5.out‚0 disk.rbd2.out‚0 disk.sda1.in‚0 proc.cpu.threads‚29 network.sent‚233879 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚0 disk.sda3.out‚ 5.7421875 disk.rbd0.in‚0$ memory_percent‚41.20221734046936 disk.dm-1.out‚ 5.7421875 disk.nvme0c0n1.in‚ 148.578125 disk.dm-0.out‚ 1773.578125 disk.sda.out‚ 5.7421875 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0 cpu‚14.965546422062378) disk./.usagePercent‚82.38512281439199% disk./.usageGB‚1227.5978660583496 disk.rbd2.in‚0 disk.nvme0n1.in‚ 148.578125! proc.memory.rssMB‚ 11266.15625 disk.dm-0.in‚ 148.578125 disk.drbd1743.in‚0 disk.drbd1743.out‚ 0.12109375 disk.rbd3.out‚0( proc.memory.percent‚17.1907901763916 network.recv‚59339 disk.rbd4.out‚ 0.06298828125 disk.sda1.out‚0" disk.nvme0c0n1.out‚ 1773.578125 disk.rbd1.in‚0* proc.memory.availableMB‚38533.71484375 disk.rbd5.in‚0 disk.sda2.out‚0 disk.nvme0n1.out‚ 1773.578125 disk.sda3.in‚0‡0cÓ0å+ _wandbruntime‚133  _runtime‚133L•mÃæ½ & lr-AdamW/pg1‚7.705883759106523e-06 & lr-AdamW/pg2‚7.705883759106523e-06 trainer/global_step‚299 % _timestamp‚1.7858039441688735e+09 _runtime‚ 133.44217112 _step‚20¿GüÇçÁ & lr-AdamW/pg1‚7.705883759106523e-06 & lr-AdamW/pg2‚7.705883759106523e-06 trainer/global_step‚299 % _timestamp‚1.7858039441688735e+09 _runtime‚ 133.44217112 _step‚20G¥90è+ _wandbruntime‚133  _runtime‚133 ¼— é‰O RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚860748 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 270799512064 _runtime‚ 133.446557102 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 263573164544 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 5289524030940 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1997479 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚204™—ÇW4245 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 262927657468 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1997477 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 5438758886912 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5154571582464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5154554543104 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚860524 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 263013460592 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚226 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1183495 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 5418232176640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5171796 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5418127707648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1997479 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 5418127707648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5154571582464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5026596373472 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3174317 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3164661 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1183497 trainer/global_step‚299 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5026613412832 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5162138 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 263573164544 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 263660594176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 263660594176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5154554543104 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3174317 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚224 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 5438914464768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3164661 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 5418232176640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 5289524030940 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1997479 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2044245 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 262927657468 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1997477 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 5438758886912 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5154571582464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5154554543104 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚860524 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 263013460592 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚226 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1183495 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 5418232176640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5171796 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5418127707648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1997479 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 5418127707648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5154571582464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5026596373472 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3174317 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31646612Z.‚0ë+ _wandbruntime‚133  _runtime‚133Py™&ì“M XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 264540220928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5307141676482 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚299 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5188976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5185191395328 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3175184 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5179304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5185166491648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 263805130690 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5171721306112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1187434 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 271844564480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 5171721306112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5436174090240 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 5436278566400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3175184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2004120 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3184854 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _step‚22 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 NDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2050790 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2051045 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 271720188928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5043336545792 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 264452784128 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1187432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚863358 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5188976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚863611 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7858039445061347e+09ábï™òí—M JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5307141676482 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚299 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5188976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5185191395328 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3175184 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5179304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5185166491648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 263805130690 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5171721306112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1187434 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 271844564480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 5171721306112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5436174090240 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 5436278566400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3175184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_byëH¡r«tes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2004120 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3184854 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _step‚22 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 NDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2050790 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2051045 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 271720188928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5043336545792 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 264452784128 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1187432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚863358 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5188976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚863611 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7858039445061347e+09 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 264540220928‚z•0î+ _wandbruntime‚133  _runtime‚133H*9züïö /loss/FBt1/detection_loss‚14.756907007572325 /loss/FBt2/detection_loss‚20.130538356931584 0loss/FBt2/assignment_loss‚2.3004445025795386 trainer/global_step‚299 0loss/FBt1/assignment_loss‚2.2902013063430786 /loss/FRt2/detection_loss‚3.8565622084243323 %loss/total_loss‚76.01468295694997 $ _timestamp‚1.785803944507402e+09 /loss/FRt1/assignment_loss‚5.520500707626343 /loss/FRt2/assignment_loss‚5.786056995391846 1loss/SRqqt1/assignment_loss‚4.531456218316005 1loss/SRqqt1/detection_loss‚3.2498136209324002 epoch‚0 _runtime‚ 133.780596043 _step‚23 1loss/SRqqt2/assignment_loss‚4.509600043296814 /loss/FRt1/detection_loss‚4.9549602380820685 0loss/SRqqt2/detection_loss‚4.127641751453644M¢ÉÈðú 0loss/FBt1/assignment_loss‚2.2902013063430786 /loss/FRt2/detection_loss‚3.8565622084243323 %loss/total_loss‚76.01468295694997 $ _timestamp‚1.785803944507402e+09 /loss/FRt1/assignment_loss‚5.520500707626343 /loss/FRt2/assignment_loss‚5.786056995391846 1loss/SRqqt1/assignment_loss‚4.531456218316005 1loss/SRqqt1/detection_loss‚3.2498136209324002 epoch‚0 _runtime‚ 133.780596043 _step‚23 1loss/SRqqt2/assignment_loss‚4.509600043296814 /loss/FRt1/detection_loss‚4.9549602380820685 0loss/SRqqt2/detection_loss‚4.127641751453644 /loss/FBt1/detection_loss‚14.756907007572325 /loss/FBt2/detection_loss‚20.130538356931584 0loss/FBt2/assignment_loss‚2.3004445025795386 trainer/global_step‚299`&ÂR›ñ‚(: ©éÄÓ¾ñžÝ gpu.0.gpu‚35.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚153.623' gpu.0.enforcedPowerLimitWatts‚180.0' gpu.0.powerPercent‚85.3461111111111 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mËWò‚(:Ì ©éÄÓä‡ßÞ disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0 disk.drbd1743.in‚0 disk.rbd2.out‚0) disk./.usagePercent‚82.38514687910022 disk.nvme0n1.out‚ 1773.96875 disk.rbd0.in‚0 disk.rbd4.out‚ 0.06298828125 disk.rbd5.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 6.43359375 disk.sda1.in‚0% memory_percent‚41.203153133392334 cpu‚14.907746926810859 network.sent‚278631% disk./.usageGB‚1227.5982246398926 disk.dm-0.out‚ 1773.96875 disk.rbd4.in‚0 disk.sda2.in‚0 disk.rbd1.out‚0 disk.dm-0.in‚ 160.68359375 disk.sda.in‚0 disk.rbd5.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 160.68359375! disk.nvme0c0n1.out‚ 1773.96875 disk.nvme0n1.in‚ 160.68359375) proc.memory.availableMB‚ 38533.1015625( proc.memory.percent‚17.1907901763916 network.recv‚62076 disk.rbd0.out‚0 disk.dm-1.out‚ 6.43359375! proc.memory.rssMB‚ 11266.15625 proc.cpu.threads‚29 disk.sda3.out‚ 6.43359375 disk.drbd1743.out‚0.140625ýsøzœó‚(:‘ ¸éÄÓ¬ä¼Û gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚147.588' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚81.99333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0• <dô‚(:Ù ¸éÄÓÀ¿ìÞ disk.rbd5.in‚0 cpu‚14.853164777564961 disk.dm-1.out‚7.015625 disk.nvme0n1.in‚ 160.68359375% disk.nvme0n1.out‚1836.91357421875 disk.sda3.in‚0 disk.rbd4.out‚ 0.06298828125 disk.sda1.out‚0 network.sent‚323333" disk.nvme0c0n1.in‚ 160.68359375 disk.sda.out‚7.015625 disk.rbd5.out‚0 disk.rbd2.in‚0! proc.memory.rssMB‚ 11266.15625 disk.sda2.out‚0' disk.nvme0c0n1.out‚1836.91357421875 disk.rbd0.out‚0 network.recv‚64346) disk./.usagePercent‚82.39353598998974 disk.dm-0.in‚ 160.68359375 disk.drbd1743.out‚ 0.1484375% disk./.usageGB‚1227.7232284545898 disk.sda2.in‚0 disk.rbd3.in‚0" disk.dm-0.out‚1836.91357421875 disk.rbd4.in‚0 disk.sda1.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.sda.in‚0 disk.drbd1743.in‚0* proc.memory.availableMB‚38532.97265625 disk.sda3.out‚7.015625 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.rbd2.out‚0$ memory_percent‚41.20334982872009( proc.memory.percent‚17.1907901763916 proc.cpu.threads‚29ç €T0õ+ _wandbruntime‚155  _runtime‚155ÿ’ÐÄö¾ _step‚24 & lr-AdamW/pg1‚8.994493083371828e-06 & lr-AdamW/pg2‚8.994493083371828e-06 trainer/global_step‚349 % _timestamp‚1.7858039660836844e+09 _runtime‚ 155.357019133ø¥ÔüÈ÷ & lr-AdamW/pg2‚8.994493083371828e-06 trainer/global_step‚349 % _timestamp‚1.7858039660836844e+09 _runtime‚ 155.357019133 _step‚24 & lr-AdamW/pg1‚8.994493083371828e-06ýzF¾0ø+ _wandbruntime‚155  _runtime‚155T£¦M‘'ù‹O OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2329629 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 307656729600 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2329627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 307569299968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1004333 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6028340118528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2329627 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 6170421110448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5863622028832 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 6170523952932 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 6012892693504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 306816120976 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 6012909732864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6028315214848 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 6320566462464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3702319 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 307569299968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 _runtime‚ 155.361576237 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2329629 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6320461993472 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6344296672768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3692663 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6344460639232 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1380347 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1380345 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3692663 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 6320461993472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚349s(Ö•'úO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 6012892693504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 306816120976 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 6012909732864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6028315214848 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 6320566462464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3702319 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 307569299968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 _runtime‚ 155.361576237 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2329629 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6320461993472 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6344296672768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3692663 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6344460639232 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1380347 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1380345 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3692663 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 6320461993472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚349 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2329629 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 307656729600 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2329627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 307569299968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1004333 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6028340118528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2329627 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 6170421110448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5863622028832 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 6170523952932 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484RgÍ0û+ _wandbruntime‚155  _runtime‚155Ù¶#… ü”M J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2391465 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1384282 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6362405772288 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2336272 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1384284 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5880345161792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 trainer/global_step‚349 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2336272 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6039456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5880362201152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _step‚26 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6045547235328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F:°_´qDeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6049128 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 308536356352 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 308536356352 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6049128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3712856 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 6030076495872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 6030059456512 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 6030076495872 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2336270 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 6030059456512 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 307693594198 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2336270 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 316883440640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6039456 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 6188038755990 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 6188141598586 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 308448919552 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1007424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 308448919552 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1007424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2391465 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1384282 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6362405772288 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2336272 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1384284 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5880345161792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 trainer/global_step‚349 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2336272 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6039456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5880362201152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _step‚26 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6045547235328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6049128 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 308536356352 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 308536356352 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6049128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3712856 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 6030076495872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 6030059456512 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 6030076495872 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2336270 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 6030059456512 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 307693594198 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2336270 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 316883440640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6039456 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 6188038755990 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136âG!0þ+ _wandbruntime‚155  _runtime‚155˜¶Üøÿò % _timestamp‚1.7858039664595015e+09 _runtime‚ 155.73261048 /loss/FBt1/assignment_loss‚2.203961319393582 0loss/FBt2/assignment_loss‚2.1831590638441196 $loss/total_loss‚80.4200034361514 /loss/FRt2/assignment_loss‚5.816478321853193 1loss/SRqqt1/assignment_loss‚4.556439270052993 1loss/SRqqt2/assignment_loss‚4.526268200701978 1loss/SRqqt1/detection_loss‚2.8973755954127562 trainer/global_step‚349 _step‚27 0loss/SRqqt2/detection_loss‚4.048802179923976 /loss/FBt2/detection_loss‚22.565514504032976 epoch‚0 /loss/FRt1/assignment_loss‚5.606653633117676 -loss/FRt1/detection_loss‚4.59209805727005 /loss/FRt2/detection_loss‚3.3701747310103722 /loss/FBt1/detection_loss‚18.053078559537727€Ç¾ü€ö /loss/FRt2/assignment_loss‚5.816478321853193 1loss/SRqqt1/assignment_loss‚4.556439270052993 1loss/SRqqt2/assignment_loss‚4.526268200701978 1loss/SRqqt1/detection_loss‚2.8973755954127562 trainer/global_step‚349 _step‚27 0loss/SRqqt2/detection_loss‚4.048802179923976 /loss/FBt2/detection_loss‚22.565514504032976 epoch‚0 /loss/FRt1/assignment_loss‚5.606653633117676 -loss/FRt1/detection_loss‚4.59209805727005 /loss/FRt2/detection_loss‚3.3701747310103722 /loss/FBt1/detection_loss‚18.053078559537727 % _timestamp‚1.7858039664595015e+09 _runtime‚ 155.73261048 /loss/FBt1/assignment_loss‚2.203961319393582 0loss/FBt2/assignment_loss‚2.1831590638441196 $loss/total_loss‚80.4200034361514»ö¯íœ‚(:‘ ÇéÄÓƶ¹Û gpu.0.gpu‚14.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚140.837' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚78.24277777777777 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00Ä?+Q‚‚(:Æ ÇéÄÓ¬‹ùÝ disk.dm-1.in‚0 disk.drbd1743.out‚0.15625 network.recv‚65789 disk.nvme0n1.in‚161.5" disk.dm-0.out‚1837.21435546875 disk.rbd4.out‚ 0.06298828125 disk.sda1.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚1837.21435546875 disk.dm-0.in‚161.5 disk.rbd5.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd3.in‚0! proc.memory.rssMB‚ 11266.15625 proc.cpu.threads‚29' disk.nvme0c0n1.out‚1837.21435546875 disk.rbd5.in‚0 disk.dm-1.out‚ 7.8046875 disk.drbd1743.in‚0 disk.rbd2.in‚0( proc.memory.percent‚17.1907901763916 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 7.8046875 disk.rbd0.out‚0* proc.memory.availableMB‚38532.70703125 network.sent‚326035 disk.nvme0c0n1.in‚161.5 disk.sda.out‚ 7.8046875% disk./.usageGB‚1227.7254905700684 disk.rbd2.out‚0 cpu‚14.804316131710035) disk./.usagePercent‚82.39368780245753 disk.sda.in‚0 disk.rbd4.in‚0% memory_percent‚41.203755140304565à.˜0ƒ+ _wandbruntime‚175  _runtime‚175)u€˜Å„¿ trainer/global_step‚399 % _timestamp‚1.7858039861710706e+09 _runtime‚ 175.44436837 _step‚28 ' lr-AdamW/pg1‚1.0283102407637132e-05 ' lr-AdamW/pg2‚1.0283102407637132e-05È? É…Ã ' lr-AdamW/pg1‚1.0283102407637132e-05 ' lr-AdamW/pg2‚1.0283102407637132e-05 trainer/global_step‚399 % _timestamp‚1.7858039861710706e+09 _runtime‚ 175.44436837 _step‚28¾ÀÂ0†+ _wandbruntime‚175  _runtime‚175ºÓV‘'‡‹O HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 6871247883264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1577195 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7222796279296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2661779 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1577197 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7222796279296 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 351565435392 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2725603 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4220665 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7222900748288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6892100 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 trainer/global_step‚399 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 6700630644832 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 351652865024 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 361294622720 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4230321 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7249832906240 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6888671054848 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7051318189956aOAÊbˆO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2661779 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7222900748288 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1148408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 351652865024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 350704584484 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 % _timestamp‚1.7858039861720808e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 351565435392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 60ƒ˜ Y3 DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2661779 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1577197 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7222796279296 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 351565435392 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2725603 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4220665 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7222900748288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6892100 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 trainer/global_step‚399 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 6700630644832 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 351652865024 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 361294622720 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4230321 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7249832906240 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6888671054848 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7051318189956 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 6871247883264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1577195 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7222796279296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 H8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2732643 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6909280 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6909280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7267940817920 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7240842661888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7240947138048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2732382 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 175.781323764 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 352445054976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4231188 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 352532491776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 351582057706 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6899608 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1581132 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚399 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 352532491776 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4231188 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4240858 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1151250 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6905878171648 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 352445054976 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7268098485760 _step‚30 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7068935835498 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2668420 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 6888414646272 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6899608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 6888414646272 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2668420 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 6717370817152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.785803986507693e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 351667860942 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4240858 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1151509 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1581134 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 6888397606912 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2668422 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 6888397606912Ò\“bž&‹˜M QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7240842661888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7240947138048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2732382 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 175.781323764 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 352445054976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4231188 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 352532491776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 351582057706 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6899608 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1581132 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚399 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 352532491776 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4231188 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4240858 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1151250 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6905878171648 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 352445054976 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7268098485760 _step‚30 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7068935835498 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2668420 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 6888414646272 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6899608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 6888414646272 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2668420 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 6717370817152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.785803986507693e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 351667860942 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4240858 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1151509 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1581134 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 6888397606912 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2668422 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 6888397606912 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2732643 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6909280 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6909280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7267940817920Þ 0Œ+ _wandbruntime‚175  _runtime‚175ÒüЊùó .loss/FRt1/detection_loss‚6.626318961381912 /loss/FRt2/detection_loss‚4.0232809328220105 1loss/SRqqt2/detection_loss‚3.7639685811644252 .loss/FBt2/detection_loss‚17.20300606773658 trainer/global_step‚399 % _timestamp‚1.7858039865089016e+09 1loss/SRqqt1/assignment_loss‚4.519129578883831 1loss/SRqqt2/assignment_loss‚4.567669792608782 0loss/SRqqt1/detection_loss‚3.265467517794325 %loss/total_loss‚70.16287524772326 0loss/FRt1/assignment_loss‚5.4852420183328485 /loss/FBt2/assignment_loss‚2.122876909646121 /loss/FBt1/detection_loss‚10.517416883508364 _runtime‚ 175.78230775 /loss/FRt2/assignment_loss‚5.890905634923414 /loss/FBt1/assignment_loss‚2.177592368920644 epoch‚0 _step‚31ÌŒýŽ÷ /loss/FRt2/assignment_loss‚5.890905634923414 /loss/FBt1/assignment_loss‚2.177592368920644 epoch‚0 _step‚31 .loss/FRt1/detection_loss‚6.626318961381912 /loss/FRt2/detection_loss‚4.0232809328220105 1loss/SRqqt2/detection_loss‚3.7639685811644252 .loss/FBt2/detection_loss‚17.20300606773658 trainer/global_step‚399 % _timestamp‚1.7858039865089016e+09 1loss/SRqqt1/assignment_loss‚4.519129578883831 1loss/SRqqt2/assignment_loss‚4.567669792608782 0loss/SRqqt1/detection_loss‚3.265467517794325 %loss/total_loss‚70.16287524772326 0loss/FRt1/assignment_loss‚5.4852420183328485 /loss/FBt2/assignment_loss‚2.122876909646121 /loss/FBt1/detection_loss‚10.517416883508364 _runtime‚ 175.78230775Ó=òO›‚(: ÖéÄÓ•ºïÝ gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚110.14' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚61.18888888888889 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0át-M‚(: ÖéÄÓàëæß disk.sda.out‚8.390625 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.rbd1.in‚0* proc.memory.availableMB‚38532.55078125 cpu‚14.763847889871956 disk.sda3.in‚0 disk.sda1.out‚0$ memory_percent‚41.20399355888367 proc.cpu.threads‚29 disk.rbd4.in‚0 disk.drbd1743.out‚ 0.18359375 disk.sda1.in‚0 disk.rbd5.in‚0 disk.dm-1.out‚8.390625 disk.sda2.out‚0 network.sent‚370964 network.recv‚68229% disk./.usageGB‚1227.7239112854004 disk.sda2.in‚0 disk.rbd1.out‚0 disk.nvme0n1.in‚161.5$ disk.nvme0n1.out‚1838.6357421875 disk.sda3.out‚8.390625 disk.rbd0.out‚0 disk.drbd1743.in‚0! proc.memory.rssMB‚ 11266.15625) disk./.usagePercent‚82.39358181533837 disk.rbd0.in‚0 disk.sda.in‚0 disk.rbd4.out‚ 0.06298828125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚161.5& disk.nvme0c0n1.out‚1838.6357421875 disk.rbd5.out‚0 disk.rbd2.out‚0( proc.memory.percent‚17.1907901763916 disk.dm-0.in‚161.5! disk.dm-0.out‚1838.6357421875Ó€›‘‚(: åéÄÓÛáÉÛ gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚38.0 gpu.0.powerWatts‚115.29' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚64.05000000000001 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôk–K’‚(:À åéÄÓ†‚óÞ! disk.dm-0.out‚1901.1787109375 disk.rbd2.in‚0 network.sent‚415552% disk./.usageGB‚1227.7239723205566 disk.rbd4.in‚0 disk.rbd5.in‚0 disk.rbd2.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda3.out‚ 9.21484375 disk.rbd1.out‚0 disk.dm-0.in‚161.5 disk.sda2.out‚0$ disk.nvme0n1.out‚1901.1787109375 disk.sda3.in‚0 disk.rbd0.in‚0$ memory_percent‚41.20445251464844$ proc.memory.availableMB‚38532.25 disk.sda.in‚0 disk.rbd5.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚161.5 disk.rbd0.out‚0) disk./.usagePercent‚82.39358591145891! proc.memory.rssMB‚ 11266.15625 proc.cpu.threads‚29 disk.sda1.out‚0 disk.nvme0c0n1.in‚161.5( proc.memory.percent‚17.1907901763916 network.recv‚70108 disk.dm-1.out‚ 9.21484375 disk.drbd1743.in‚0 disk.drbd1743.out‚0.203125& disk.nvme0c0n1.out‚1901.1787109375 disk.sda.out‚ 9.21484375 disk.rbd4.out‚ 0.06298828125 cpu‚14.7260508246635951Ç“+ _wandbruntime‚19 1mg7  _runtime‚197FÂ,Å”¿ $ _timestamp‚1.785804008175759e+09 _runtime‚ 197.449024692 _step‚32 ' lr-AdamW/pg1‚1.1571711731902438e-05 ' lr-AdamW/pg2‚1.1571711731902438e-05 trainer/global_step‚449Yfd‹É•Ã ' lr-AdamW/pg2‚1.1571711731902438e-05 trainer/global_step‚449 $ _timestamp‚1.785804008175759e+09 _runtime‚ 197.449024692 _step‚32 ' lr-AdamW/pg1‚1.1571711731902438e-05 Ÿh&³0–+ _wandbruntime‚197  _runtime‚197p¢<¦'—ŠO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 7749026894848 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 406493060608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1774047 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 $ _timestamp‚1.785804008176774e+09 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8125235034112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7932215269464 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7752252 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1291609 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8125235034112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7729586033664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2993929 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7537622221472 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 394593047992 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 _step‚33 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3065656 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8125130565120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8125130565120 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4758323 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1774045 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8155389281280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0¶ ”'˜ŽO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 trainer/global_step‚449 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8125130565120 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4758323 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1774045 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8155389281280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 7749026894848 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 406493060608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1774047 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 $ _timestamp‚1.785804008176774e+09 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8125235034112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7932215269464 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7752252 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1291609 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8125235034112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7729586033664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2993929 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7537622221472 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 394593047992 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 _step‚33 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3065656 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8125130565120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4759190 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8173657834496 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7554362393792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚449 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7746752796672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4759190 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3072184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7746735757312 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7759760 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 N6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 407266155008 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 395470521214 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7769432 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4768860 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 7766234011648 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7949832915006 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 197.795498875 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 396441190400 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _step‚34 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7759760 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8143281423872 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7769432 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 407398919168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7949935757602 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 396441190400 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1294451 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 7766258915328 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 $ _timestamp‚1.785804008521694e+09½EIY›˜M Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8143176947712 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 396528627200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMoniõ~­%œtor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 395470521214 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7769432 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4768860 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 7766234011648 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7949832915006 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 197.795498875 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 396441190400 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _step‚34 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7759760 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8143281423872 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7769432 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 407398919168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7949935757602 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 396441190400 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1294451 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 7766258915328 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 $ _timestamp‚1.785804008521694e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4759190 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8173657834496 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7554362393792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚449 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7746752796672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4759190 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3072184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7746735757312 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7759760 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 N6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 407266155008 Bÿžù 1loss/SRqqt1/assignment_loss‚4.588752520935876 1loss/SRqqt2/assignment_loss‚4.547503942175757 /loss/FBt2/assignment_loss‚2.020067165879642 /loss/FBt2/detection_loss‚22.100719878778737 % _timestamp‚1.7858040085229962e+09 /loss/FRt1/assignment_loss‚5.548717769224252 0loss/FBt1/assignment_loss‚1.9878429133316566 1loss/SRqqt1/detection_loss‚3.0001213656339263 _runtime‚ 197.796153143 /loss/FRt2/assignment_loss‚5.954356842570835 0loss/SRqqt2/detection_loss‚4.211973780129529 %loss/total_loss‚72.83851676877627 .loss/FRt1/detection_loss‚5.058088737192438 /loss/FRt2/detection_loss‚3.1818597287767463 /loss/FBt1/detection_loss‚10.638512124146882 epoch‚0 trainer/global_step‚449 _step‚35#R"&‘Ÿ‚(:† ôéÄÓŽƒÜ gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚153.441' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚85.245 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(­˜¿K ‚(:À ôéÄÓχÐß( proc.memory.percent‚17.1907901763916 cpu‚14.693295408031805 network.recv‚70480 disk.drbd1743.in‚0 disk.rbd3.in‚0" disk.nvme0n1.out‚ 1904.61328125$ memory_percent‚41.20534658432007 proc.cpu.threads‚29 disk.sda3.out‚ 9.76171875 disk.rbd4.out‚ 0.06298828125 disk.sda1.in‚0$ disk.nvme0c0n1.out‚ 1904.61328125! proc.memory.rssMB‚ 11266.15625) disk./.usagePercent‚82.39359435970755 disk.rbd0.in‚0 disk.dm-0.in‚161.5 disk.rbd5.out‚0 disk.dm-1.in‚0 disk.rbd2.out‚0 disk.nvme0n1.in‚161.5) proc.memory.availableMB‚ 38531.6640625 disk.rbd4.in‚0% disk./.usageGB‚1227.7240982055664 disk.sda3.in‚0 disk.sda.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚161.5 disk.rbd1.out‚0 disk.dm-0.out‚ 1904.61328125 disk.rbd5.in‚0 disk.dm-1.out‚ 9.76171875 disk.drbd1743.out‚ 0.2109375 disk.rbd1.in‚0 network.sent‚415924 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 9.76171875 disk.sda1.out‚0 disk.sda2.in‚0ÚÅ 90¡+ _wandbruntime‚217  _runtime‚217÷ìŽÆ¢À trainer/global_step‚499 % _timestamp‚1.7858040280491931e+09 _runtime‚ 217.322522458 _step‚36 ' lr-AdamW/pg1‚1.2860321056167744e-05 ' lr-AdamW/pg2‚1.2860321056167744e-05è]s¶Ê£Ä trainer/global_step‚499 % _timestamp‚1.7858040280491931e+09 _runtime‚ 217.322522458 _step‚36 ' lr-AdamW/pg1‚1.2860321056167744e-05 ' lr-AdamW/pg2‚1.2860321056167744e-05$jåà.0¤+ _wandbruntime‚217  _runtime‚217v‘'¥‹O WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 8587907144704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1970897 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9061063236096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 438481511500 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 _runtime‚ 217.326538039 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8612404 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 439557706240 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 9027569319936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1433856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8609382734848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 8813112348972 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 438567314624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1970895 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5286325 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3405035 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 trainer/global_step‚499 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5286325 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 9027464850944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8602746 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 439557706240 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 451516534784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 9027464850944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 8587907144704 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3404751 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9060899269632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 439645135872 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3326077 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 8374647876832 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5286325 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 9027464850944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8602746 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 439557706240 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 451516534784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 9027464850944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 8587907144704 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3404751 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9060899269632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 439645135872 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3326077 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 8374647876832 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 9027569319936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1433856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8609382734848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 8813112348972 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 438567314624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1970895 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5286325 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3405035 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121%Åøk•0§+ _wandbruntime‚217  _runtime‚217…éW†¶¨”M A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _step‚38 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 8605090947072 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 440437325824 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8629584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 \KDeviceStatsMonitor.on_train_batch_ÌÿqVä"end/inactive_split_bytes.small_pool.freed‚ 452418435584 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 8605073907712 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5296862 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 8605073907712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1974834 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8626589851648 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3332720 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 8830729994514 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8629584 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5287192 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3411552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 8391371009792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 8605090947072 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3332722 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 9045615709696 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 440524762624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1436962 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1436720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 8391388049152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9079168052224 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.785804028387181e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 8830832837110 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 N6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8629584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 452418435584 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 8605073907712 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5296862 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 8605073907712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1974834 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8626589851648 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3332720 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 8830729994514 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8629584 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5287192 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3411552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 8391371009792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 8605090947072 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3332722 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 9045615709696 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 440524762624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1436962 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1436720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 8391388049152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9079168052224 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.785804028387181e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 8830832837110 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 N6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 496873106432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3658229 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 9469738574848 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9211656492832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9211639453472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3658227 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 9446262334464 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1577561 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9472556 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 482369975008 trainer/global_step‚549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 9446262334464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3658229 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 9694112270964 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3745308 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1577298 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3658227 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2167745 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5814327 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 _step‚41 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 9929799136768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9966636584960 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 496742432256 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5804671 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3745043 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9462898 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 9446245295104 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 483641271296 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 9929799136768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 9694009428480 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 9929903605760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176V£Ý”´O O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocatiöí”Ò'ons.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5804671 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3745043 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9462898 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 9446245295104 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 483641271296 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 9929799136768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 9694009428480 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 9929903605760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5814327 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 9929903605760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 9469763478528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7858040503651557e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 496873106432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3658229 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 9469738574848 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9211656492832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9211639453472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3658227 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 9446262334464 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1577561 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9472556 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 482369975008 trainer/global_step‚549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 9446262334464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3658229 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 9694112270964 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3745308 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1577298 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3658227 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2167745 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5814327 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 _step‚41 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 9929799136768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9966636584960 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 496742432256 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251)…hí0µ+ _wandbruntime‚239  _runtime‚239ûš&¶”M D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _runtime‚ 239.979497917 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9480064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 484520898048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 497646987264 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9489736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3664870 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 9228379625792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9486970595328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 9228396665152 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 497777654272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 9463412058112 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 9947949995520 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 9463429097472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5824864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9489736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 9711627074022 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9984592678912 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 9711729916618 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 484433461248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3751815 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 trainer/global_step‚549 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9480064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3664870 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 483333251466 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 $ _timestamp‚1.785804050705819e+09 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3752069 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5824864 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 484520898048 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 484520898048 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _runtime‚ 239.979497917 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9480064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 484520898048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 497646987264 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9489736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3664870 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 9228379625792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9486970595328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 9228396665152 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 497777654272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 9463412058112 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 9947949995520 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 9463429097472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5824864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9489736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 9711627074022 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9984592678912 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 9711729916618 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 484433461248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3751815 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008*ŸG0¸+ _wandbruntime‚239  _runtime‚239Â*­Êú¹ô 0loss/FBt2/assignment_loss‚1.4879439142015245 1loss/SRqqt1/detection_loss‚3.2565527219795487 epoch‚0 trainer/global_step‚549 .loss/FRt1/detection_loss‚6.908075061440468 1loss/SRqqt2/detection_loss‚3.7404782248749617 %loss/total_loss‚91.77960250160521 1loss/SRqqt1/assignment_loss‚4.401299509029944 2loss/SRqqt2/assignment_loss‚4.4491799502145675 0loss/FBt1/assignment_loss‚1.3171912245452404 /loss/FRt2/detection_loss‚3.1178325991564932 ,loss/FBt1/detection_loss‚9.4602826843402 .loss/FBt2/detection_loss‚41.96085916293992 % _timestamp‚1.7858040507074249e+09 _runtime‚ 239.980537447 _step‚43 /loss/FRt1/assignment_loss‚5.674685373306274 /loss/FRt2/assignment_loss‚6.005222075576082øbþºø /loss/FRt1/assignment_loss‚5.674685373306274 /loss/FRt2/assignment_loss‚6.005222075576082 0loss/FBt2/assignment_loss‚1.4879439142015245 1loss/SRqqt1/detection_loss‚3.2565527219795487 epoch‚0 trainer/global_step‚549 .loss/FRt1/detection_loss‚6.908075061440468 1loss/SRqqt2/detection_loss‚3.7404782248749617 %loss/total_loss‚91.77960250160521 1loss/SRqqt1/assignment_loss‚4.401299509029944 2loss/SRqqt2/assignment_loss‚4.4491799502145675 0loss/FBt1/assignment_loss‚1.3171912245452404 /loss/FRt2/detection_loss‚3.1178325991564932 ,loss/FBt1/detection_loss‚9.4602826843402 .loss/FBt2/detection_loss‚41.96085916293992 % _timestamp‚1.7858040507074249e+09 _runtime‚ 239.980537447 _step‚43+UºÏd‘»‚(:† ’êÄÓ¡áÈÛ gpu.0.gpu‚42.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚144.333' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚80.185 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]K!!Ÿ¼‚(:Ç ’êÄÓ”½ûÞ) disk./.usagePercent‚82.41038000570508 disk.dm-1.out‚ 11.1484375 disk.sda1.out‚0& disk.nvme0c0n1.out‚1967.5302734375 disk.rbd1.in‚0 disk.rbd0.out‚0$ memory_percent‚41.20588302612305( proc.memory.percent‚17.1908438205719% disk./.usageGB‚1227.9742164611816 disk.drbd1743.out‚ 0.24609375 disk.rbd2.in‚0 disk.nvme0c0n1.in‚161.5 disk.sda3.in‚0 cpu‚14.631093532157934 proc.cpu.threads‚29 network.sent‚417136 disk.rbd5.out‚0$ disk.nvme0n1.out‚1967.5302734375 disk.sda.out‚ 11.1484375 disk.dm-1.in‚0 disk.rbd2.out‚0 disk.sda1.in‚0 disk.sda.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚161.5 network.recv‚72336 disk.dm-0.in‚161.5 disk.rbd4.out‚ 0.06298828125 disk.sda2.in‚0 disk.rbd1.out‚0 disk.rbd0.in‚0$ proc.memory.rssMB‚11266.19140625! disk.dm-0.out‚1967.5302734u{Ë’³375 disk.rbd4.in‚0 disk.rbd5.in‚0 disk.drbd1743.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 11.1484375& proc.memory.availableMB‚ 38531.3125‰gºaœ½‚(:‘ ¡êÄÓ•ï»Û gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚153.512' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚85.28444444444445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_H“V¾‚(:Ë ¡êÄÓ×ïÝ disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚41.20594263076782$ proc.memory.rssMB‚11266.19140625$ disk.nvme0n1.out‚1967.8115234375 disk.rbd5.out‚0 disk.rbd3.out‚0 network.recv‚73403) disk./.usagePercent‚82.41038384581809 disk.rbd5.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚161.5% disk./.usageGB‚1227.9742736816406 disk.rbd1.in‚0 disk.sda.in‚0 disk.rbd4.in‚0 disk.dm-1.out‚ 12.03515625 disk.sda1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚29 disk.dm-0.in‚161.5 disk.rbd4.out‚ 0.06298828125 disk.drbd1743.in‚0 disk.sda.out‚ 12.03515625 disk.rbd2.in‚0& disk.nvme0c0n1.out‚1967.8115234375) proc.memory.availableMB‚ 38531.2734375 disk.drbd1743.out‚0.265625 disk.rbd2.out‚0( proc.memory.percent‚17.1908438205719 disk.rbd1.out‚0! disk.dm-0.out‚1967.8115234375 cpu‚14.610394058917812 network.sent‚417703 disk.nvme0n1.in‚161.5 disk.sda3.in‚0 disk.sda3.out‚ 12.03515625¶ 0¿+ _wandbruntime‚259  _runtime‚259¸ð¶ŽÅÀ¿ ' lr-AdamW/pg1‚1.5437539704698353e-05 ' lr-AdamW/pg2‚1.5437539704698353e-05 trainer/global_step‚599 $ _timestamp‚1.785804070045305e+09 _runtime‚ 259.318582471 _step‚44\ý úÉÁà _runtime‚ 259.318582471 _step‚44 ' lr-AdamW/pg1‚1.5437539704698353e-05 ' lr-AdamW/pg2‚1.5437539704698353e-05 trainer/global_step‚599 $ _timestamp‚1.785804070045305e+09,Ç;0Â+ _wandbruntime‚259  _runtime‚259.æ*K¥'ßO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 527637406720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10872174580736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6342329 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 527549977088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 trainer/global_step‚599 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚10304583445504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚ 259.323545499 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚10323050 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚10575009350472 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚10832133422592 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10330094414848 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚10574906507988 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6332673 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2364597 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 541926685184 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 526258438516 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4085894 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚10048648069472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10872021100032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7858040700462234e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚10304583445504 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 527549977088 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10832133422592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360ÀŽC©'Ä£O QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 trainer/global_step‚599 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚10304583445504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚ 259.323545499 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚10323050 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚10575009350472 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚10832133422592 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10330094414848 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚10574906507988 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6332673 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2364597 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 541926685184 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 526258438516 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4085894 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚10048648069472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10872021100032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7858040700462234e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚10304583445504 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 527549977088 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10832133422592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 527637406720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10872174580736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6342329 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 527549977088-½ Ø0Å+ _wandbruntime‚259  _runtime‚259žÖØý®&ƨM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 $ _timestamp‚1.785804070383394e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚10321767247872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10340216 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4092409 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2368532 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚10065405281152 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 527221714974 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6343196 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 527135911738 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚10850284281344 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10340216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10349888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _step‚46 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10349888 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6352866 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2368534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚10592626996126 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚10321750208512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10347326435328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6343196 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10890128307200 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3997022 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3997020 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 528429596672 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚10850179805184 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚10850284281344 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 528517033472 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 _runtime‚ 259.657018386 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6352866 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0sÀ/íǬM XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 528517033472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.½âÅ%small_pool.allocated‚ 542961636864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3997022 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6343196 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 527135911738 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚10850284281344 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10340216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10349888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _step‚46 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10349888 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6352866 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2368534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚10592626996126 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚10321750208512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10347326435328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6343196 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10890128307200 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3997022 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3997020 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 528429596672 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚10850179805184 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚10850284281344 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 528517033472 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 _runtime‚ 259.657018386 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6352866 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 $ _timestamp‚1.785804070383394e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚10321767247872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10340216 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4092409 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2368532 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚10065405281152 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 527221714974 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 B&XÌ‚(:Í °êÄÓήÚÞ disk.rbd3.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 12.49609375 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.drbd1743.in‚0 disk.drbd1743.out‚ 0.2734375$ memory_percent‚41.20666980743408) disk./.usagePercent‚82.41038512585575 disk.dm-0.in‚161.5 disk.sda.out‚ 12.49609375( proc.memory.availableMB‚ 38530.796875$ proc.memory.rssMB‚11266.19140625 disk.rbd2.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚2030.34814453125" disk.dm-0.out‚2030.34814453125 disk.rbd1.out‚0% disk.nvme0n1.out‚2030.34814453125$ disk./.usageGB‚1227.974292755127 disk.sda2.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚161.5( proc.memory.percent‚17.1908438205719 disk.rbd4.in‚0 disk.rbd5.out‚0 disk.rbd2.out‚0 disk.sda1.in‚0 network.sent‚554930 network.recv‚79330 disk.rbd4.out‚ 0.06298828125 disk.rbd5.in‚0 disk.dm-1.out‚ 12.49609375 disk.nvme0c0n1.in‚161.5 disk.rbd1.in‚0 cpu‚14.586854389174349 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda2.out‚0 proc.cpu.threads‚29@·¯!0Í+ _wandbruntime‚281  _runtime‚281•$G›ÆÎÀ % _timestamp‚1.7858040923410606e+09 _runtime‚ 281.614363254 _step‚48 ' lr-AdamW/pg1‚1.6726149028963658e-05 ' lr-AdamW/pg2‚1.6726149028963658e-05 trainer/global_step‚649ý®òÝÊÏÄ ' lr-AdamW/pg2‚1.6726149028963658e-05 trainer/global_step‚649 % _timestamp‚1.7858040923410606e+09 _runtime‚ 281.614363254 _step‚48 ' lr-AdamW/pg1‚1.6726149028963658e-050OiLê0Ð+ _wandbruntime‚281  _runtime‚281àeò¥'ÑŸO NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4426566 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚11777510671872 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4322527 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6860675 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11183202 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4322529 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6870331 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11734467708416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1865121 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2561445 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚49 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 571633542144 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4322527 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 % _timestamp‚1.7858040923421125e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11162921595904 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 587060417024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2561447 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1865366 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11734467708416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4322529 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚ 281.618576386 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11190450254848 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11455906429980 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 571633542144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4426813 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 587193188352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11162938635264 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 570232705148 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚10885673724832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656í96w:$Ò£O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚10885673724832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4426566 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚11777510671872 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4322527 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6860675 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11183202 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4322529 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6870331 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11734467708416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1865121 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2561445 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚49 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 571633542144 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4322527 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 % _timestamp‚1.7858040923421125e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11162921595904 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 587060417024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2561447 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1865366 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11734467708416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4322529 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚ 281.618576386 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11190450254848 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11455906429980 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 571633542144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4426813 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 587193188352 G/¿öoADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11162938635264 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 570232705148 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1028424841àtÇQ0Ó+ _wandbruntime‚281  _runtime‚2817ÓÁ¯&Ô©M ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚11180105398272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4433538 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6871198 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚11795779601920 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 571110178482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11200368 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2565382 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 571024375246 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚11180105398272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 588097326592 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6880868 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11180088358912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚11207682275328 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 587968756736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚10902396857792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6871198 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11207657371648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4433296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚11752514091008 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 572425732096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4329170 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4329170 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1868154 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚ 281.953730427 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1867914 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11210040 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 572425732096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11200368 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚11473524075634 % _timestamp‚1.7858040926800823e+09 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 572513168896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11210040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 trainer/global_step‚649 _step‚50 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2565384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 572513168896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚10902413897152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0`ëÓ{³&Õ­M MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6880868 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11180088358912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚11207682275328 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 587968756736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚10902396857792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6871198 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11207657371648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4433296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚11752514091008 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 572425732096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4329170 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4329170 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1868154 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚ 281.953730427 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1867914 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11210040 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 572425732096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11200368 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚11473524075634 % _timestamp‚1.7858040926800823e+09 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 572513168896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11210040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 trainer/global_step‚649 _step‚50 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2565384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 572513168896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚10902413897152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚11180105398272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4433538 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6871198 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚11795779601920 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 571110178482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11200368 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2565382 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 571024375246 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚11180105398272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 588097326592 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1028425962PT+F0Ö+ _wandbruntime‚281  _runtime‚281D«³Ÿü×ö 0loss/FBt2/assignment_loss‚0.8213592107806887 0loss/SRqqt1/detection_loss‚3.714082416470932 /loss/FBt1/detection_loss‚6.7210008106449095 /loss/FBt2/detection_loss‚26.510674151991093 epoch‚0 _runtime‚ 281.954476036 _step‚51 /loss/FRt1/assignment_loss‚5.483347709362324 /loss/FRt2/assignment_loss‚5.972662342404856 2loss/SRqqt2/assignment_loss‚3.6823441419708596 0loss/FBt1/assignment_loss‚0.8939116761088371 .loss/FRt1/detection_loss‚5.208590714748089 /loss/FRt2/detection_loss‚3.3551046803448963 0loss/SRqqt2/detection_loss‚3.407606558685892 %loss/total_loss‚69.34670899766162 1loss/SRqqt1/assignment_loss‚3.576024584148241 trainer/global_step‚649 % _timestamp‚1.7858040926813312e+09ª¥Ò„Øú 0loss/SRqqt1/detection_loss‚3.714082416470932 /loss/FBt1/detection_loss‚6.7210008106449095 /loss/FBt2/detection_loss‚26.510674151991093 epoch‚0 _runtime‚ 281.954476036 _step‚51 /loss/FRt1/assignment_loss‚5.483347709362324 /loss/FRt2/assignment_loss‚5.972662342404856 2loss/SRqqt2/assignment_loss‚3.6823441419708596 0loss/FBt1/assignment_loss‚0.8939116761088371 .loss/FRt1/detection_loss‚5.208590714748089 /loss/FRt2/detection_loss‚3.3551046803448963 0loss/SRqqt2/detection_loss‚3.407606558685892 %loss/total_loss‚69.34670899766162 1loss/SRqqt1/assignment_loss‚3.576024584148241 trainer/global_step‚649 % _timestamp‚1.7858040926813312e+09 0loss/FBt2/assignment_loss‚0.82135921078068873r‘Ù‚(:† ¿êÄÓ»óžÛ gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚154.629' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚85.905 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0謈YÚ‚(:Î ¿êÄÓ¶‹òÝ disk.rbd1.in‚0 disk.sda.out‚ 13.3203125 disk.rbd4.in‚0 disk.rbd5.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.drbd1743.in‚0 disk.sda2.out‚0* proc.memory.availableMB‚38530.46484375 disk.sda3.in‚0 network.sent‚556102 network.recv‚80771 disk.rbd1.out‚0 disk.sda.in‚0 disk.rbd2.out‚0 proc.cpu.threads‚29% disk.nvme0n1.out‚2030.60595703125 disk.sda3.out‚ 13.3203125 disk.dm-0.in‚161.5" disk.dm-0.out‚2030.60595703125 disk.rbd2.in‚0 cpu‚14.562687304113114% disk./.usageGB‚1227.9742736816406 disk.drbd1743.out‚ 0.27734375) disk./.usagePercent‚82.41038384581809 disk.rbd0.in‚0$ proc.memory.rssMB‚11266.19140625 disk.nvme0n1.in‚161.5 disk.rbd3.in‚0 disk.nvme0c0n1.in‚161.5' disk.nvme0c0n1.out‚2030.60595703125 disk.rbd4.out‚ 0.06298828125 disk.sda1.out‚0$ memory_percent‚41.20717644691467( proc.memory.percent‚17.1908438205719 disk.rbd3.out‚0 disk.rbd5.out‚0 disk.dm-1.out‚ 13.3203125@œÛ‚(:‘ ÎêÄÓ¦¹ÙÛ gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚148.119' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚82.28833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#–AMÜ‚(: ÎêÄÓÙÆß network.sent‚600996 disk.sda.in‚0 disk.rbd4.out‚ 0.06298828125 disk.drbd1743.in‚0! disk.nvme0n1.out‚ 2030.9140625$ memory_percent‚41.20725393295288 disk.dm-0.out‚ 2030.9140625 disk.rbd5.out‚0 disk.sda3.out‚ 13.90234375) proc.memory.availableMB‚ 38530.4140625 cpu‚14.544484468465367 disk.rbd4.in‚0 disk.dm-1.out‚ 13.90234375 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd1.out‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚161.5 disk.rbd5.in‚0 disk.drbd1743.out‚ 0.3046875 disk.rbd2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚161.5 disk.sda3.in‚0( proc.memory.percent‚17.1908438205719) disk./.usagePercent‚82.40131196283475 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.rbd2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk./.usageGB‚1227.839096069336# disk.nvme0c0n1.out‚ 2030.9140625 disk.rbd1.in‚0 network.recv‚83051 disk.dm-0.in‚161.5 disk.sda.out‚ 13.90234375$ proc.memory.rssMB‚11266.19140625 proc.cpu.threads‚29F«Ë0Ý+ _wandbruntime‚301  _runtime‚301¾ >ÆÞÀ trainer/global_step‚699 % _timestamp‚1.7858041121002483e+09 _runtime‚ 301.373536398 _step‚52 ' lr-AdamW/pg1‚1.8014758353228962e-05 ' lr-AdamW/pg2‚1.8014758353228962e-05¬ü*ÊßÄ ' lr-AdamW/pg2‚1.8014758353228962e-05 trainer/global_step‚699 % _timestamp‚1.7858041121002483e+09 _runtime‚ 301.373536398 _step‚52 ' lr-AdamW/pg1‚1.8014758353228962e-054c'8B0à+ _wandbruntime‚301  _runtime‚301øÕl á¤O YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 614035365532 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7398333 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚12336803509488 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7388677 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12683207156736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12050830998528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 _runtime‚301.37856785500003 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 632237095424 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2007271 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2758295 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4654679 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2758297 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 615629677568 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚12021259746304 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 615629677568 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚12021276785664 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2007030 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7398333 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚12021276785664 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12683043190272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11722682340832 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4654677 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚12636906463232 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12636801994240 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12636906463232 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12636801994240 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12636906463232 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2758295 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4654679 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2758297 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 615629677568 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚12021259746304 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6156296775685Ì:³ù0ã+ _wandbruntime‚301  _runtime‚301ØHΩ¯&ä©M UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2762232 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12701158450176 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 616421867520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚12354421155142 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7399200 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12068038115328 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12060520 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4772012 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12701309826560 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 614912838754 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 633145238528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚54 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚12038443548672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4661322 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 614998641990 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚12038426509312 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 616421867520 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12060520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4661320 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2010025 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12068013211648 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚699 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚12354318312546 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚12038426509312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4772259 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2762234 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4661320 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 616509304320 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚12654952852992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11739422513152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12070192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12654848376832 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚12038443548672 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12070192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12654848376832 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚12038443548672 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 616421867520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚12354421155142 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7399200 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12068038115328 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12060520 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4772012 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12701309826560 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 614912838754 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 633145238528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚54 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚12038443548672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4661322 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 614998641990 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚12038426509312 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 616421867520 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12060520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4661320 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2010025 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12068013211648 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚699 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚12354318312546 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚12038426509312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4772259 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2762234 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4661320 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 616509304320 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚12654952852992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11739422513152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDevÕàxxmiceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 F7q0î+ _wandbruntime‚323  _runtime‚323O›U¥'ïŸO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12903506 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚12879614936064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 677582898688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 trainer/global_step‚749 % _timestamp‚1.7858041340448098e+09 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12911186838528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚12559690956832 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12913164 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2150965 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5106112 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12911161934848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7916679 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4986829 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚13217597746512 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚12559673917472 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 658009632164 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚12879597896704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 657923829040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 _step‚57 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 659625812992 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4986827 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 677448030208 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚13539136280064 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚13539240749056 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7916679 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13588609965056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4986829 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚13539136280064 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7926335 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚13539240749056 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2955147 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚ 323.318629208 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392_¼ê·©'ð£O JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2150965 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5106112 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12911161934848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7916679 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4986829 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚13217597746512 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚12559673917472 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 658009632164 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚12879597896704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 657923829040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 _step‚57 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 659625812992 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4986827 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 677448030208 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚13539136280064 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚13539240749056 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7916679 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13588609965056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4986829 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚13539136280064 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7926335 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚13539240749056 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2955147 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚ 323.318629208 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12903506 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚12879614936064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 677582898688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 trainer/global_step‚749 % _timestamp‚1.7858041340448098e+09 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12911186838528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚12559690956832 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12913164 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚09Àö&h0ñ+ _wandbruntime‚323  _runtime‚323N¬NÛ±ò©M ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚13557287138816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5112653 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2153812 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚13235215392054 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4993472 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13606727832064 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7858041343819246e+09 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7927202 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13606877111296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 _runtime‚ 323.655529415 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 678483155968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2153571 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12928369051648 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 678358780416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 660418002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2959084 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚12576431129152 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5112896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4993470 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12920672 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 660418002944 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7927202 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚12576414089792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 _step‚58 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12930344 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12930344 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7936872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2959082 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7936872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚12896781699072 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚12896764659712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4993472 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13235318234650 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 658887105498ÙÑMõ³&ó­M MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4993472 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13235318234650 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 658887105498 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚13557287138816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5112653 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2153812 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚13235215392054 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4993472 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13606727832064 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7858041343819246e+09 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7927202 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13606877111296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 _runtime‚ 323.655529415 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 678483155968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2153571 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12928369051648 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 678358780416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 660418002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2959084 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚12576431129152 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5112896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4993470 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12920672 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 660418002944 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7927202 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚12576414089792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 _step‚58 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12930344 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12930344 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7936872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2959082 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7936872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚12896781699072 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚12896764659712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800:pÖÊ0ô+ _wandbruntime‚323  _runtime‚323™?Òýõ÷ /loss/FRt2/assignment_loss‚5.901399527277265 0loss/FBt2/assignment_loss‚0.7151694903771083 /loss/FBt2/detection_loss‚24.404794983069102 1loss/SRqqt1/assignment_loss‚2.729601798579097 2loss/SRqqt2/assignment_loss‚2.9587288205052764 /loss/FRt2/detection_loss‚3.0958883967250586 0loss/SRqqt2/detection_loss‚4.225873064039878 _runtime‚ 323.656347509 _step‚59 .loss/FRt1/detection_loss‚5.271926674577925 0loss/SRqqt1/detection_loss‚2.238014541740995 &loss/total_loss‚63.484482189627435 trainer/global_step‚749 1loss/FBt1/assignment_loss‚0.38985044327942103 .loss/FBt1/detection_loss‚6.068838324813313 epoch‚0 % _timestamp‚1.7858041343832102e+09 /loss/FRt1/assignment_loss‚5.484396124642993»¢jIöû &loss/total_loss‚63.484482189627435 trainer/global_step‚749 1loss/FBt1/assignment_loss‚0.38985044327942103 .loss/FBt1/detection_loss‚6.068838324813313 epoch‚0 % _timestamp‚1.7858041343832102e+09 /loss/FRt1/assignment_loss‚5.484396124642993 /loss/FRt2/assignment_loss‚5.901399527277265 0loss/FBt2/assignment_loss‚0.7151694903771083 /loss/FBt2/detection_loss‚24.404794983069102 1loss/SRqqt1/assignment_loss‚2.729601798579097 2loss/SRqqt2/assignment_loss‚2.9587288205052764 /loss/FRt2/detection_loss‚3.0958883967250586 0loss/SRqqt2/detection_loss‚4.225873064039878 _runtime‚ 323.656347509 _step‚59 .loss/FRt1/detection_loss‚5.271926674577925 0loss/SRqqt1/detection_loss‚2.238014541740995;pðœ÷‚(:‘ ìêÄÓÔÈÃÛ gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚38.0 gpu.0.powerWatts‚102.971' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚57.20611111111111 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G³9#Yø‚(:Î ìêÄÓã¡èÞ network.sent‚647899 disk.sda.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.drbd1743.in‚0 disk.rbd2.out‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚161.5$ proc.memory.rssMB‚11266.19140625! disk.dm-0.out‚2093.8505859375 disk.rbd4.in‚0 disk.rbd4.out‚ 0.06298828125& disk.nvme0c0n1.out‚2093.8505859375 disk.sda3.out‚ 15.14453125$ memory_percent‚41.20672345161438 cpu‚14.508324363591358 proc.cpu.threads‚29) disk./.usagePercent‚82.40971387410099 disk.drbd1743.out‚ 0.33203125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0* proc.memory.availableMB‚38530.76171875( proc.memory.percent‚17.1908438205719% disk./.usageGB‚1227.9642906188965 disk.rbd5.out‚0 disk.nvme0c0n1.in‚161.5 disk.rbd5.in‚0 disk.rbd1.out‚0 network.recv‚86971 disk.dm-0.in‚161.5 disk.dm-1.out‚ 15.14453125$ disk.nvme0n1.out‚2093.8505859375 disk.sda.out‚ 15.144531259›uG0ù+ _wandbruntime‚343  _runtime‚343߇õŸÄú¾ trainer/global_step‚799 % _timestamp‚1.7858041537772331e+09 _runtime‚ 343.050531691 _step‚60 & lr-AdamW/pg1‚2.059197700175957e-05 & lr-AdamW/pg2‚2.059197700175957e-05 Ø tÈû _runtime‚ 343.050531691 _step‚60 & lr-AdamW/pg1‚2.059197700175957e-05 & lr-AdamW/pg2‚2.059197700175957e-05 trainer/global_step‚799 % _timestamp‚1.7858041537772331e+09<‰»™P0ü+ _wandbruntime‚343  _runtime‚343Èánæ¥'ýŸO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚13737953086464 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5318979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14441470565888 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14441470565888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8454337 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 703534518784 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13773316 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚61 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14098597668504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8444681 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8454337 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚14098494826020 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 703621948416 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 722809232896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8444681 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚14494351911424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13737936047104 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 703621948416 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5446512 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13771542678528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13771517774848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5318977 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5318979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _runtime‚ 343.055599241 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2294515 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚13396699572832 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13396682533472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5446243 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13763658 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13737953086464çN™S@þ£O [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13396682533472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5446243 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13763658 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13737953086464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚13737953086464 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5318979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14441470565888 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14441470565888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8454337 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 703534518784 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13773316 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚61 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14098597668504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8444681 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8454337 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚14098494826020 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 703621948416 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 722809232896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8444681 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚14494351911424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13737936047104 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 703621948416 PDDeviceStatsMonitor.on_trai{‡%üi n_batch_start/inactive_split.all.allocated‚5446512 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13771542678528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13771517774848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5318977 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5318979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _runtime‚ 343.055599241 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2294515 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚13396699572832=&¦ë0ÿ+ _wandbruntime‚343  _runtime‚343¦¬›¯&€©M TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 704414138368 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚13413422705792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8455204 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2297378 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8464874 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8455204 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14116112471562 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5325622 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5453312 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚14512302049792 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13790496 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5325620 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚13788724891648 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5325620 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14459621424640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 723712019456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 704414138368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚13755102810112 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 704501575168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 702689765770 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 704501575168 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5453071 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3155932 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2297139 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚62 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13780824 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3155934 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚13413439745152 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7858041541575844e+09 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚13755119849472 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 702775569006 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14459621424640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14116215314158 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13790496 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13755102810112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 _runtime‚ 343.431332298 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14459621424640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 723712019456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 704414138368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚13755102810112 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 704501575168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 702689765770 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 704501575168 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5453071 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3155932 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2297139 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚62 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13780824 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3155934 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚13413439745152 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7858041541575844e+09 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚13755119849472 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 702775569006 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14459621424640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14116215314158 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13790496 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13755102810112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 _runtime‚ 343.431332298 FïJ!B0‚+ _wandbruntime‚343  _runtime‚343ÆLã4÷ƒñ -loss/FRt1/detection_loss‚5.82740220688937 1loss/SRqqt2/detection_loss‚3.8516621409290575 trainer/global_step‚799 .loss/FRt2/detection_loss‚3.025558177543723 epoch‚0 _runtime‚ 343.432331413 _step‚63 0loss/SRqqt1/detection_loss‚2.082267766220238 .loss/FBt1/detection_loss‚8.337865938729264 0loss/SRqqt2/assignment_loss‚2.91892573803286 1loss/FBt1/assignment_loss‚0.18850161002150603 .loss/FBt2/detection_loss‚61.80269655585289 &loss/total_loss‚103.06394208521507 $ _timestamp‚1.785804154159125e+09 /loss/FRt1/assignment_loss‚5.572850327742727 /loss/FRt2/assignment_loss‚5.953865362250286 0loss/SRqqt1/assignment_loss‚2.66127003596081 0loss/FBt2/assignment_loss‚0.8410762250423431¬·)û„õ 0loss/SRqqt2/assignment_loss‚2.91892573803286 1loss/FBt1/assignment_loss‚0.18850161002150603 .loss/FBt2/detection_loss‚61.80269655585289 &loss/total_loss‚103.06394208521507 $ _timestamp‚1.785804154159125e+09 /loss/FRt1/assignment_loss‚5.572850327742727 /loss/FRt2/assignment_loss‚5.953865362250286 0loss/SRqqt1/assignment_loss‚2.66127003596081 0loss/FBt2/assignment_loss‚0.8410762250423431 -loss/FRt1/detection_loss‚5.82740220688937 1loss/SRqqt2/detection_loss‚3.8516621409290575 trainer/global_step‚799 .loss/FRt2/detection_loss‚3.025558177543723 epoch‚0 _runtime‚ 343.432331413 _step‚63 0loss/SRqqt1/detection_loss‚2.082267766220238 .loss/FBt1/detection_loss‚8.337865938729264?ZЀœ…‚(:‘ ûêÄÓØ÷«Û gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚38.0 gpu.0.powerWatts‚118.252' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚65.69555555555556 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õjj F†‚(:» ûêÄÓ²²½ß disk.rbd0.out‚0 disk.sda.in‚0 disk.rbd4.in‚0 disk.sda2.in‚0% disk.nvme0c0n1.out‚2094.869140625( proc.memory.percent‚17.1908438205719 disk.sda3.in‚0 disk.sda.out‚16.125 disk.rbd4.out‚ 0.06298828125 disk.rbd5.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚161.5 disk.rbd0.in‚0 disk.dm-0.out‚2094.869140625 disk.dm-1.out‚16.125 disk.rbd2.out‚0 disk.sda2.out‚0 network.recv‚89523 disk.nvme0n1.in‚161.5) proc.memory.availableMB‚ 38530.2578125$ proc.memory.rssMB‚11266.19140625 network.sent‚693201 disk.rbd3.out‚0 disk.rbd1.out‚0) disk./.usagePercent‚82.40971413010851 disk.dm-1.in‚0$ memory_percent‚41.20749235153198 cpu‚14.492136519065788% disk./.usageGB‚1227.9642944335938 disk.drbd1743.out‚ 0.33984375 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda3.out‚16.125 disk.dm-0.in‚161.5 disk.rbd5.out‚0 disk.drbd1743.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚29# disk.nvme0n1.out‚2094.869140625‘Htœ‡‚(:‘ ŠëÄÓÄäÂÜ gpu.0.gpu‚22.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚38.0 gpu.0.powerWatts‚146.761' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚81.53388888888888 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ(ÐÞUˆ‚(:Ê ŠëÄÓ¸¼Îß disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚161.5 disk.sda3.out‚ 16.51171875 disk.sda.out‚ 16.51171875 network.sent‚737514% disk./.usageGB‚1227.9642181396484 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚161.5 disk.rbd4.out‚ 0.06298828125 disk.rbd5.in‚0 disk.dm-1.out‚ 16.51171875 disk.sda2.in‚0% disk.nvme0c0n1.out‚2157.337890625* proc.memory.availableMB‚38529.85546875$ proc.memory.rssMB‚11266.19140625 cpu‚14.477029335400111 disk.rbd1.out‚0 disk.dm-0.out‚2157.337890625 disk.sda.in‚0 disk.drbd1743.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚0 network.recv‚91909) disk./.usagePercent‚82.40970900995784 disk.rbd0.out‚0 disk.dm-0.in‚161.5( proc.memory.percent‚17.1908438205719 proc.cpu.threads‚29# disk.nvme0n1.out‚2157.337890625 disk.sda3.in‚0 disk.sda1.out‚0$ memory_percent‚41.20810627937317 disk.rbd4.in‚0 disk.rbd5.out‚0 disk.drbd1743.out‚ 0.3671875ºLj0‰+ _wandbruntime‚365  _runtime‚365,p‘§ÆŠÀ trainer/global_step‚849 % _timestamp‚1.7858041761261625e+09 _runtime‚ 365.399517284 _step‚64 ' lr-AdamW/pg1‚2.1880586326024876e-05 ' lr-AdamW/pg2‚2.1880586326024876e-05vèhÞÊ‹Ä ' lr-AdamW/pg2‚2.1880586326024876e-05 trainer/global_step‚849 % _timestamp‚1.7858041761261625e+09 _runtime‚ 365.399517284 _step‚64 ' lr-AdamW/pg1‚2.1880586326024876e-05@rš+Ÿ0Œ+ _wandbruntime‚365  _runtime‚365¾NT°ŸO JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14596291236864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2438774 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15399916882432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5651129 % _timestamp‚1.7858041761271875e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15343804851712 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8982339 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5651127 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14979494748012 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8972683 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 747618083840 TGDeviceStatsMonitor.on_tOf'*õrain_batch_start/active_bytes.small_pool.current‚87429632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚14631898518528 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15343909320704 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 747530654208 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15343804851712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5787619 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 365.403553396 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14233708188832 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8972683 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14596274197504 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5787875 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 767881398272 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8982339 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5651129 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15399755013120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 768018363904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚15343909320704 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2439028 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚14979391905528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14596291236864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14596274197504 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15343804851712 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8982339 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5651127 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14979494748012 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8972683 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 747618083840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚14631898518528 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15343909320704 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 747530654208 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15343804851712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5787619 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 365.403553396 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14233708188832 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8972683 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14596274197504 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5787875 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 767881398272 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130A݇ $0+ _wandbruntime‚365  _runtime‚365õ,e1®&¨M H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15418022624256 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 768784224768 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14640976 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5657772 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14613457999872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 746664032514 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14613440960512 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 768916988928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14997112393666 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15361955710464 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14997009551070 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14650648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14250431321792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14649080731648 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5657772 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2441601 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3352784 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 748497710592 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14640976 trainer/global_step‚849 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 748410273792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5794383 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14613457999872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8992876 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15361955710464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15417864956416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14649105635328 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8992876 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 $ _timestamp‚1.785804176465997e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2441844 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14250448361152 _step‚66 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 748497710592 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5794628 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5657770 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8983206 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14613440960512 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 365.739645123 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚146506485@”sa‘¬M JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5794628 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5657770 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8983206 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14613440960512 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 365.739645123 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14650648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15418022624256 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 768784224768 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14640976 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5657772 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14613457999872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 746664032514 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14613440960512 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 768916988928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14997112393666 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15361955710464 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14997009551070 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14650648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14250431321792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14649080731648 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5657772 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2441601 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3352784 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 748497710592 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14640976 trainer/global_step‚849 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 748410273792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5794383 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14613457999872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8992876 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S@Devig}NQ ceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15361955710464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15417864956416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14649105635328 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8992876 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 $ _timestamp‚1.785804176465997e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2441844 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14250448361152 _step‚66 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 748497710592 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16246139137536 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 789675022688 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 % _timestamp‚1.7858041962392359e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3545697 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15492254358528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 789589219564 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15454612347904 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚15070716804832 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚16246243606528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚15454612347904 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5983277 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚16305511479808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚15454629387264 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15860288985036 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3545695 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15070699765472 _runtime‚ 385.516572559 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 791526789632 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9500685 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚16305351707648 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15493620 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15483962 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 791526789632 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚16246243606528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15454629387264 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5983279 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0d4@.©'œ£O WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15483962 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 791526789632 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚16246243606528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15454629387264 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5983279 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5983277 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 791614219264 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9510341 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6128161 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5983279 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚15860391827520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9510341 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6128420 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 791614219264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 813122252800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 trainer/global_step‚899 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16246139137536 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 789675022688 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 % _timestamp‚1.7858041962392359e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3545697 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15492254358528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 789589219564 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15454612347904 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚15070716804832 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚16246243606528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚15454612347904 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5983277 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚16305511479808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚15454629387264 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15860288985036 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3545695 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15070699765472 _runtime‚ 385.516572559 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 791526789632 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9500685 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚16305351707648 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15493620 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0Eë^ P0+ _wandbruntime‚385  _runtime‚385Ž³&P¾ž¨M @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6134948 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚16323620713472 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 792493846016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚15087456977152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3549634 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 814159238144 _step‚70 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15510800 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9511208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15087439937792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 % _timestamp‚1.7858041966361897e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚16264289996288 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚16264185520128 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚16323467239936 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 trainer/global_step‚899 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9511208 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5989922 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6135204 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3549632 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5989920 G:DeviceStatsMonitor.on_train_batch_end/allocation.+@¹§ðall.freed‚15501128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5989920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15471796150272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15471779110912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9520878 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 790552496022 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 790466692786 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15501128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9520878 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15509436571648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2585570 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15509461475328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 _runtime‚ 385.91007523 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5989922 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 792493846016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 792406409216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 814030668288Ðײ&Ÿ¬M REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15878009473174 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9520878 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15509436571648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2585570 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15509461475328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 _runtime‚ 385.91007523 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5989922 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 792493846016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 792406409216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 814030668288 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6134948 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚16323620713472 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 792493846016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚15087456977152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3549634 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 814159238144 _step‚70 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15510800 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9511208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15087439937792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 % _timestamp‚1.7858041966361897e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚16264289996288 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚16264185520128 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚16323467239936 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 trainer/global_step‚899 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9511208 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5989922 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6135204 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3549632 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5989920 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15501128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5989920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15471796150272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15471779110912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9520878 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 790552496022 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 790466692786 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15501128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 835610354688 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16312950498304 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16353772 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17148473423360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚15907725420832 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚17148577892352 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16312950498304 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2727646 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 $ _timestamp‚1.785804218158037e+09 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6469946 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16352585294848 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 833563486196 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 _step‚73 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚949Þ²ý]¬¦O OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6315429 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10028687 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 835610354688 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16312950498304 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16353772 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 @:DeviceStatsMonitor.o âíOn_train_batch_start/num_oom_rejections‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17148473423360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚15907725420832 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚17148577892352 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16312950498304 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2727646 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 $ _timestamp‚1.785804218158037e+09 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6469946 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16352585294848 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 833563486196 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 _step‚73 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚949 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 835522925056 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15907708381472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3742545 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16312967537664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6315427 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2727401 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 858364686336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16312967537664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚17148577892352 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10038343 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10028687 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 835610354688 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16370952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15924448553792 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 834355156294 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚15924465593152 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16330117261312 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 836489981440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.785804218498563e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6476741 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2730506 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16369817315328 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16361280 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10048880 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3746482 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 859268962816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6322070 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 859401726976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10039210 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚17166624282112 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16330134300672 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 836489981440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6476990 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3746484 trainer/global_step‚949 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _runtime‚ 407.772105758 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670Kܶ&¯°M FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16369817315328 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16361280 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10048880 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3746482 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 859268962816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6322070 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 859401726976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10039210 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚17166624282112 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16330134300672 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 836489981440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6476990 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3746484 trainer/global_step‚949 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _runtime‚ 407.772105758 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16370952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15924448553792 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 834355156294 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚15924465593152 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16330117261312 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 836489981440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.785804218498563e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6476741 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2730506 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121JaБ[0°+ _wandbruntime‚407  _runtime‚407ep×"ü±ö 1loss/SRqqt1/assignment_loss‚2.373503507886614 1loss/FBt1/assignment_loss‚0.10011970611481831 %loss/total_loss‚70.39804436414998 % _timestamp‚1.7858042184998019e+09 _runtime‚ 407.772903505 epoch‚0 /loss/FRt1/assignment_loss‚5.464187708057341 0loss/FBt2/assignment_loss‚0.6630632792684165 .loss/FRt1/detection_loss‚5.298784690313652 1loss/SRqqt1/detection_loss‚2.2131927985492177 .loss/FBt1/detection_loss‚6.752051957727201 trainer/global_step‚949 _step‚75 1loss/SRqqt2/assignment_loss‚2.688329301318344 /loss/FRt2/detection_loss‚3.1587911501666097 1loss/SRqqt2/detection_loss‚3.1946133849667193 .loss/FBt2/detection_loss‚32.89121628484943 /loss/FRt2/assignment_loss‚5.600190594931629Ø¡r²ú /loss/FRt2/detection_loss‚3.1587911501666097 1loss/SRqqt2/detection_loss‚3.1946133849667193 .loss/FBt2/detection_loss‚32.89121628484943 /loss/FRt2/assignment_loss‚5.600190594931629 1loss/SRqqt1/assignment_loss‚2.373503507886614 1loss/FBt1/assignment_loss‚0.10011970611481831 %loss/total_loss‚70.39804436414998 % _timestamp‚1.7858042184998019e+09 _runtime‚ 407.772903505 epoch‚0 /loss/FRt1/assignment_loss‚5.464187708057341 0loss/FBt2/assignment_loss‚0.6630632792684165 .loss/FRt1/detection_loss‚5.298784690313652 1loss/SRqqt1/detection_loss‚2.2131927985492177 .loss/FBt1/detection_loss‚6.752051957727201 trainer/global_step‚949 _step‚75 1loss/SRqqt2/assignment_loss‚2.688329301318344K~D?š³‚(: ÆëÄÓ†é¶Û gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚147.83' gpu.0.enforcedPowerLimitWatts‚180.0' gpu.0.powerPercent‚82.1277777777778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾ÈzW´‚(:Ì ÆëÄÓ뿈ß% disk./.usageGB‚1228.2146644592285 disk.nvme0c0n1.in‚161.5 disk.rbd1.out‚0" disk.dm-0.out‚2220.68408203125 disk.dm-1.in‚0 disk.dm-1.out‚ 19.828125 disk.sda2.in‚0 cpu‚14.424834558345562 disk.drbd1743.out‚ 0.42578125 disk.sda1.in‚0 disk.rbd3.out‚0) proc.memory.availableMB‚ 38529.9296875$ proc.memory.rssMB‚11266.19140625% disk.nvme0n1.out‚2220.68408203125 disk.rbd4.out‚ 0.06298828125 disk.rbd2.in‚0% memory_percent‚41.207993030548096 network.sent‚833116 disk.dm-0.in‚161.5( proc.memory.percent‚17.1908438205719 proc.cpu.threads‚29 disk.rbd2.out‚0' disk.nvme0c0n1.out‚2220.68408203125 disk.sda3.out‚ 19.828125 disk.rbd5.in‚0 disk.drbd1743.in‚0 network.recv‚100013) disk./.usagePercent‚82.42651667260331 disk.nvme0n1.in‚161.5 disk.sda3.in‚0 disk.rbd5.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 19.828125 disk.rbd4.in‚0€o¥g0µ+ _wandbruntime‚428  _runtime‚4288Ÿ™Æ¶À ' lr-AdamW/pg1‚2.5746414298820792e-05 ' lr-AdamW/pg2‚2.5746414298820792e-05 trainer/global_step‚999 % _timestamp‚1.7858042390550804e+09 _runtime‚ 428.328488569 _step‚76ñA©?Ê·Ä ' lr-AdamW/pg1‚2.5746414298820792e-05 ' lr-AdamW/pg2‚2.5746414298820792e-05 trainer/global_step‚999 % _timestamp‚1.7858042390550804e+09 _runtime‚ 428.328488569 _step‚76Lšc0¸+ _wandbruntime‚428  _runtime‚428E±Çª ¹£O LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚17212966038528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 trainer/global_step‚999 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚17171288648704 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚18050912178176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚18116653054464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18050807709184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 879519060480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17204266 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6647579 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 879606490112 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peakfú;¼ÿ‚538 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10566345 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6809594 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚17212941134848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 877451949704 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚17622185986536 _step‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 877366146580 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18050912178176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚17171305688064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18050807709184 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 903550050304 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 903550050304 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚18116653054464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18050807709184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 879519060480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17204266 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6647579 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 879606490112 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10566345 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6809594 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚17212941134848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 877451949704 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚17622185986536 _step‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 877366146580 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18050912178176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚17171305688064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130M5è60»+ _wandbruntime‚428  _runtime‚428ˆmý—³&¼­M UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3943334 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚999 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚17188455411712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17231104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 880486116864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚17639803632190 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 904456489472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚17188472451072 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17231104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16761457169792 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 880398680064 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17221432 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16761474209152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 878329423038 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 880398680064 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚18134762408960 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10576882 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10567212 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18068958567936 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚18068958567936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _runtime‚ 428.664319411 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17221432 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3943332 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10567212 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6654222 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚17188455411712 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6816613 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6816361 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2873279 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6654220 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 880486116864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 _step‚78 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10576882 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7858042393907075e+09e–,·N½±M M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10576882 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10567212 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18068958567936 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚18068958567936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _runtime‚ 428.664319411 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17221432 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3943332 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10567212 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6654222 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚17188455411712 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6816613 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6816361 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2873279 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6654220 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 880486116864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 _step‚78 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10576882 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7858042393907075e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3943334 ZMDeviceStatsMonitor.on_train_batch_eno;àid/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚999 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚17188455411712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17231104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 880486116864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚17639803632190 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 904456489472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚17188472451072 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17231104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16761457169792 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 880398680064 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17221432 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16761474209152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 878329423038 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 880398680064 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚18134762408960N…%!0¾+ _wandbruntime‚428  _runtime‚428· dû¿õ -loss/FRt1/assignment_loss‚5.7077225957598 1loss/SRqqt1/assignment_loss‚2.393111704525195 1loss/FBt1/assignment_loss‚0.17232512317657953 1loss/SRqqt1/detection_loss‚2.4233634720009785 .loss/FBt1/detection_loss‚5.434081163126985 _runtime‚ 428.665089254 2loss/SRqqt2/assignment_loss‚2.7881390348486943 .loss/FRt1/detection_loss‚5.837064480675118 _step‚79 0loss/FBt2/assignment_loss‚0.5454934740749499 1loss/SRqqt2/detection_loss‚2.8491415195936045 /loss/FBt2/detection_loss‚29.959524725874264 /loss/FRt2/assignment_loss‚5.475064424758262 /loss/FRt2/detection_loss‚3.6220470660544457 %loss/total_loss‚67.20707878446888 trainer/global_step‚999 $ _timestamp‚1.785804239391961e+09 epoch‚0sàZ¤ÿÀù /loss/FRt2/assignment_loss‚5.475064424758262 /loss/FRt2/detection_loss‚3.6220470660544457 %loss/total_loss‚67.20707878446888 trainer/global_step‚999 $ _timestamp‚1.785804239391961e+09 epoch‚0 -loss/FRt1/assignment_loss‚5.7077225957598 1loss/SRqqt1/assignment_loss‚2.393111704525195 1loss/FBt1/assignment_loss‚0.17232512317657953 1loss/SRqqt1/detection_loss‚2.4233634720009785 .loss/FBt1/detection_loss‚5.434081163126985 _runtime‚ 428.665089254 2loss/SRqqt2/assignment_loss‚2.7881390348486943 .loss/FRt1/detection_loss‚5.837064480675118 _step‚79 0loss/FBt2/assignment_loss‚0.5454934740749499 1loss/SRqqt2/detection_loss‚2.8491415195936045 /loss/FBt2/detection_loss‚29.959524725874264OÛŸùìœÁ‚(:‘ ÕëÄÓ®äåÛ gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚38.0 gpu.0.powerWatts‚123.112' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚68.39555555555556 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍÄÜX‚(:Í ÕëÄÓüØ€ß disk.sda3.in‚0 disk.sda3.out‚ 20.5703125 disk.sda.in‚0 disk.dm-1.out‚ 20.5703125' disk.nvme0c0n1.out‚2283.96826171875 disk.dm-0.in‚161.5 disk.rbd5.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0$ memory_percent‚41.20765924453735$ proc.memory.rssMB‚11266.19140625( proc.memory.percent‚17.1908438205719 cpu‚14.411162216800577 disk.rbd4.out‚ 0.06298828125 disk.rbd0.out‚0 disk.sda.out‚ 20.5703125 disk.drbd1743.out‚ 0.4453125 disk.rbd2.out‚0 disk.nvme0n1.in‚161.5% disk./.usageGB‚1228.2147026062012" disk.dm-0.out‚2283.96826171875) proc.memory.availableMB‚ 38530.1484375 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚0 network.sent‚878096 disk.rbd4.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚2283.96826171875 proc.cpu.threads‚29 disk.rbd5.out‚0 disk.rbd3.in‚0 disk.drbd1743.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚161.5 disk.rbd2.in‚0 network.recv‚102665) disk./.usagePercent‚82.42651923267866i‡¦^0Ã+ _wandbruntime‚449  _runtime‚449sïþ%ÇÄÁ ' lr-AdamW/pg1‚2.7035023623086093e-05 ' lr-AdamW/pg2‚2.7035023623086093e-05 trainer/global_step‚1049 % _timestamp‚1.7858042605955055e+09 _runtime‚ 449.868853328 _step‚80Ù¬¹ËÅÅ % _timestamp‚1.7858042605955055e+09 _runtime‚ 449.868853328 _step‚80 ' lr-AdamW/pg1‚2.7035023623086093e-05 ' lr-AdamW/pg2‚2.7035023623086093e-05 trainer/global_step‚1049PÙ§JI0Æ+ _wandbruntime‚449  _runtime‚449³ß½ª'ǤO IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18953141995008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 % _timestamp‚1.7858042605965097e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚17581725613472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚18503083066044 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 923602625536 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18073296974848 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 921340413212 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1049 _runtime‚ 449.873555276 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚18953246464000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚17581742652832 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚18029643838464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18073321878528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18953246464000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚18029626799104 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6979727 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚18502980223560 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18953141995008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 923515195904 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7149725 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4136245 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7149442µB)0®'ȨO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 923515195904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚18029643838464 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11094347 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11084691 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11094347 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18073296974848 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 921340413212 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1049 _runtime‚ 449.873555276 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚18953246464000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚17581742652832 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚18029643838464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18073321878528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18953246464000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚18029626799104 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6979727 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚18502980223560 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18953141995008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 923515195904 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7149725 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4136245 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7149442 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18953141995008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 % _timestamp‚1.7858042605965097e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚17581725613472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚18503083066044 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 923602625536 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚18971292853760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11095214 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18090528995328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18091256 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚18046810601472 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 924482252288 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11095214 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18090504091648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚1049 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚18046793562112 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7156208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 G:DeviceStatsMonitor.on_train_batc`V¹)h_end/active.all.allocated‚18091256 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18971292853760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3016278 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 949609828352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6986372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚18520597869102 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚17598465785792 _runtime‚ 450.204821639 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚17598482825152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11104884 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4140182 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6986370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18081584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7156462 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19040271587840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 % _timestamp‚1.7858042609311564e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3016026 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 922132083310 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 924482252288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚82 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6986372—‘䎸&ͲM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 924482252288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚82 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6986372 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚18971292853760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11095214 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18090528995328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18091256 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚18046810601472 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 924482252288 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11095214 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18090504091648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚1049 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚18046793562112 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7156208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18091256 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18971292853760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3016278 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 949609828352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6986372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚18520597869102 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚17598465785792 _runtime‚ 450.204821639 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚17598482825152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11104884 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4140182 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6986370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18081584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7156462 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19040271587840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 % _timestamp‚1.7858042609311564e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3016026 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 922132083310R ߃o0Î+ _wandbruntime‚450  _runtime‚450Ę¢ ÿÏù 1loss/SRqqt2/detection_loss‚3.7516695347585483 % _timestamp‚1.7858042609323878e+09 /loss/FRt1/assignment_loss‚5.576174020767212 2loss/SRqqt1/assignment_loss‚2.2525379277290183 1loss/SRqqt2/assignment_loss‚2.662096320228143 1loss/SRqqt1/detection_loss‚2.1741174918016886 .loss/FBt1/detection_loss‚5.096930134105213 &loss/total_loss‚59.434155794569904 epoch‚0 _runtime‚ 450.205495393 0loss/FBt2/assignment_loss‚0.6442434171835582 /loss/FBt2/detection_loss‚23.304562756667533 /loss/FRt2/assignment_loss‚5.248539447784424 .loss/FRt1/detection_loss‚5.853325098220791 .loss/FRt2/detection_loss‚2.738661336442155 trainer/global_step‚1049 _step‚83 1loss/FBt1/assignment_loss‚0.13129830888162056§BŠÐý /loss/FRt2/assignment_loss‚5.248539447784424 .loss/FRt1/detection_loss‚5.853325098220791 .loss/FRt2/detection_loss‚2.738661336442155 trainer/global_step‚1049 _step‚83 1loss/FBt1/assignment_loss‚0.13129830888162056 1loss/SRqqt2/detection_loss‚3.7516695347585483 % _timestamp‚1.7858042609323878e+09 /loss/FRt1/assignment_loss‚5.576174020767212 2loss/SRqqt1/assignment_loss‚2.2525379277290183 1loss/SRqqt2/assignment_loss‚2.662096320228143 1loss/SRqqt1/detection_loss‚2.1741174918016886 .loss/FBt1/detection_loss‚5.096930134105213 &loss/total_loss‚59.434155794569904 epoch‚0 _runtime‚ 450.205495393 0loss/FBt2/assignment_loss‚0.6442434171835582 /loss/FBt2/detection_loss‚23.304562756667533S¸HGœÑ‚(:‘ óëÄÓôǹÛ gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚152.974' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚84.98555555555555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºëd`Ò‚(:Õ óëÄӥݡÞ% disk./.usageGB‚1228.2147254943848 disk.drbd1743.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚161.5 proc.cpu.threads‚29 disk.rbd0.in‚0 disk.dm-1.out‚21.8125$ proc.memory.rssMB‚11266.19140625 disk.sda3.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚2284.69775390625 disk.sda.in‚0 disk.rbd5.in‚0 disk.rbd5.out‚0 disk.nvme0c0n1.in‚161.5) disk./.usagePercent‚82.42652076872386 disk.sda.out‚21.8125 disk.rbd4.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚2284.69775390625 cpu‚14.390844238111505 network.sent‚956817 disk.rbd4.out‚ 0.06298828125 disk.dm-1.in‚0 disk.drbd1743.out‚ 0.4609375 disk.rbd2.in‚0$ memory_percent‚41.20786786079407* proc.memory.availableMB‚38530.01171875( proc.memory.percent‚17.1908438205719 disk.rbd2.out‚ 0.1484375 disk.sda2.in‚0 disk.rbd3.out‚0 disk.rbd3.in‚0 disk.sda3.out‚21.8125 disk.sda1.in‚0' disk.nvme0c0n1.out‚2284.69775390625 disk.rbd1.out‚ 0.1484375 network.recv‚107436 disk.dm-0.in‚161.5 disk.rbd1.in‚0 ˜\0Ó+ _wandbruntime‚470  _runtime‚470˜<Œ®ÇÔÁ _step‚84 ' lr-AdamW/pg1‚2.8323632947351405e-05 ' lr-AdamW/pg2‚2.8323632947351405e-05 trainer/global_step‚1099 % _timestamp‚1.7858042811913815e+09 _runtime‚ 470.466556427Û YËÕÅ _step‚84 ' lr-AdamW/pg1‚2.8323632947351405e-05 ' lr-AdamW/pg2‚2.8323632947351405e-05 trainer/global_step‚1099 % _timestamp‚1.7858042811913815e+09 _runtime‚ 470.466556427T¼5tK0Ö+ _wandbruntime‚470  _runtime‚470^V/•ª'פO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18924570 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚18887981988864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚19855580749824 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4333097 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚19383877303068 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚18887964949504 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7490621 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11612693 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚18887964949504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚1099 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 967511331328 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19927593776640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4333095 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11612693 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚19855476280832 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18934228 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3157526 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7311879 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18933652814848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 965143073596 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 993940961792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0áÄgÈúبO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11612693 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚18887964949504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚1099 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 967511331328 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19927593776640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4333095 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11612693 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚19855476280832 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18934228 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3157526 JDDeviceStatsMonitor.on_train_batch_start/oversize_aªxk`´!llocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7311879 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18933652814848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 965143073596 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 993940961792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18924570 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚18887981988864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚19855580749824 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4333097 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚19383877303068 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚18887964949504 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7490621 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251UlT@s0Ù+ _wandbruntime‚470  _runtime‚470çÁ¯´&Ú®M F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 trainer/global_step‚1099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚18905148751872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18950884835328 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18951408 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11632886 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 968478387712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 966020546818 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18941736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3160375 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19401597791206 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚19873522663424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18435474401792 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7318522 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19401494948610 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚18905148751872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11623216 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18941736 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7318520 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18950859931648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _step‚86 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11623216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 968390950912 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19945858661888 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7497653 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7497407 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 994843159552 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚19873627139584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18951408 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7318522 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚18905131712512 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚18905131712512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚18435491441152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236žx,¸&Û²M KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18951408 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11632886 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 968478387712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 966020546818 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18941736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3160375 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19401597791206 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚19873522663424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18435474401792 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7318522 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19401494948610 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚18905148751872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11623216 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18941736 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7318520 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18950859931648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _step‚86 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11623216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 968390950912 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19945858661888 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7497653 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7497407 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 994843159552 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚19873627139584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18951408 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7318522 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚18905131712512 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚18905131712512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚18435491441152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 trainer/global_step‚1099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚18905148751872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18950884835328 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888VÜt¬d0Ü+ _wandbruntime‚470  _runtime‚470Á?¹ûÝõ /loss/FRt2/assignment_loss‚5.109171129993556 %loss/total_loss‚66.32125462759836 epoch‚0 1loss/SRqqt2/assignment_loss‚2.632227631630721 /loss/FRt2/detection_loss‚3.5258605329646278 0loss/SRqqt1/detection_loss‚2.601101313857276 1loss/SRqqt2/detection_loss‚2.8287246655457206 _runtime‚ 470.80168994 _step‚87 1loss/FBt1/assignment_loss‚0.19109415338374675 .loss/FRt1/detection_loss‚5.632936525447615 /loss/FBt2/detection_loss‚29.230869565159082 /loss/FRt1/assignment_loss‚5.474987169791913 1loss/SRqqt1/assignment_loss‚2.225891497555901 0loss/FBt2/assignment_loss‚0.6653860285878181 .loss/FBt1/detection_loss‚6.203004413680379 trainer/global_step‚1099 $ _timestamp‚1.785804281528304e+09ÍyÿÞù %loss/total_loss‚66.32125462759836 epoch‚0 1loss/SRqqt2/assignment_loss‚2.632227631630721 /loss/FRt2/detection_loss‚3.5258605329646278 0loss/SRqqt1/detection_loss‚2.601101313857276 1loss/SRqqt2/detection_loss‚2.8287246655457206 _runtime‚ 470.80168994 _step‚87 1loss/FBt1/assignment_loss‚0.19109415338374675 .loss/FRt1/detection_loss‚5.632936525447615 /loss/FBt2/detection_loss‚29.230869565159082 /loss/FRt1/assignment_loss‚5.474987169791913 1loss/SRqqt1/assignment_loss‚2.225891497555901 0loss/FBt2/assignment_loss‚0.6653860285878181 .loss/FBt1/detection_loss‚6.203004413680379 trainer/global_step‚1099 $ _timestamp‚1.785804281528304e+09 /loss/FRt2/assignment_loss‚5.109171129993556W Ya'œß‚(:‘ ‚ìÄÓûžªÜ gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚38.0 gpu.0.powerWatts‚122.822' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚68.23444444444445 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•ßlà‚(:á ‚ìÄÓÆÌ¿ß disk.sda.out‚ 22.42578125 disk.rbd5.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda3.out‚ 22.42578125* proc.memory.availableMB‚38529.48828125 disk.rbd5.out‚0 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 network.sent‚991351 disk.rbd4.in‚0 disk.dm-1.in‚0 disk.rbd2.out‚ 0.1806640625 disk.sda1.out‚0 disk.sda3.in‚0) disk./.usagePercent‚82.42651616058824! disk.dm-0.out‚2347.1669921875 disk.drbd1743.in‚0$ proc.memory.rssMB‚11266.19140625( proc.memory.percent‚17.1908438205719 disk.rbd4.out‚ 0.06298828125 disk.sda2.in‚0 disk.rbd1.out‚ 0.1484375 cpu‚14.379673834055646 disk.dm-0.in‚161.5 disk.drbd1743.out‚ 0.48828125 disk.rbd2.in‚0& disk.nvme0c0n1.out‚2347.1669921875$ disk.nvme0n1.out‚2347.1669921875 disk.nvme0c0n1.in‚161.5 network.recv‚109652$ disk./.usageGB‚1228.214656829834 disk.sda1.in‚0 disk.nvme0n1.in‚161.5 disk.rbd0.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 22.42578125$ memory_percent‚41.20866656303406 proc.cpu.threads‚29ÍùÅð0á+ _wandbruntime‚492  _runtime‚492µgþÇâÁ ' lr-AdamW/pg1‚2.9612242271616706e-05 ' lr-AdamW/pg2‚2.9612242271616706e-05 trainer/global_step‚1149 % _timestamp‚1.7858043030269608e+09 _runtime‚ 492.300289626 _step‚88U"”'ËãÅ ' lr-AdamW/pg1‚2.9612242271616706e-05 ' lr-AdamW/pg2‚2.9612242271616706e-05 trainer/global_step‚1149 % _timestamp‚1.7858043030269608e+09 _runtime‚ 492.300289626 _step‚88X}Ù)ç0ä+ _wandbruntime‚492  _runtime‚492°{Ó*å«O OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7644029 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4529947 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4529945 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_5Å_6‡%pool.freed‚ 1009031537104 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20757810566656 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚19794033558528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7644027 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7831626 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12140695 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚20757915035648 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚19746303099904 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1011594896384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚19794008654848 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1011507466752 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7644029 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 trainer/global_step‚1149 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19784722 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3301679 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1011507466752 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚19255759884832 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7644027 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12150351 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20757915035648 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19794380 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚ 492.30456285 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19255742845472 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12140695 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 % _timestamp‚1.7858043030280259e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚19746320139264 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20757810566656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _step‚89 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19746303099904 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656t£çåµ'æ¯O OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7644029 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4529947 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4529945 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1009031537104 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20757810566656 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚19794033558528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7644027 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7831626 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12140695 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚20757915035648 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚19746303099904 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1011594896384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚19794008654848 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1011507466752 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7644029 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 trainer/global_step‚1149 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19784722 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3301679 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1011507466752 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚19255759884832 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7644027 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12150351 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20757915035648 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19794380 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚ 492.30456285 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19255742845472 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12140695 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 % _timestamp‚1.7858043030280259e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚19746320139264 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20757810566656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _step‚89 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19746303099904 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656YÒÄ¢\0ç+ _wandbruntime‚492  _runtime‚492Äݬ¼&è¶M ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19763486902272 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12151218 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4533882 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚19811215771648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7858043034206917e+09 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20775856949248 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚20282392028118 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1012474523136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7650672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚19763469862912 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1009994813562 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20775961425408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19801888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1040019300864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7650670 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚19272500057152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12160888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚20282494870714 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19811560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19763469862912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚20851384351744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7650670 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1009909010326 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19811560 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19801888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1040143676416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1012387086336 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚19272483017792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12160888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7838210 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚19811240675328 trainer/global_step‚1149ÓŽ÷èµ éºM FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19801888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1040143676416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1012387086336 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚19272483017792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12160888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7838210 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚19811240675328 trainer/global_step‚1149 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19763486902272 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12151218 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4533882 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚19811215771648 >8DeviceS ‹žÅ tatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7858043034206917e+09 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20775856949248 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚20282392028118 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1012474523136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7650672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚19763469862912 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1009994813562 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20775961425408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19801888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1040019300864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7650670 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚19272500057152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12160888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚20282494870714 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19811560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19763469862912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚20851384351744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7650670 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1009909010326 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19811560Zȳd¶0ê+ _wandbruntime‚492  _runtime‚492»ªuÎüëö 2loss/SRqqt1/assignment_loss‚2.2408299351280387 1loss/FBt2/assignment_loss‚0.41993311177939174 .loss/FRt1/detection_loss‚5.446856138110161 /loss/FRt2/detection_loss‚3.0938392751807466 /loss/SRqqt1/detection_loss‚2.16483876569958 .loss/FBt2/detection_loss‚23.43605588003993 _runtime‚ 492.695475207 /loss/FRt1/assignment_loss‚5.494402162233988 /loss/FRt2/assignment_loss‚4.943970594931086 0loss/SRqqt2/detection_loss‚3.252529955536668 trainer/global_step‚1149 % _timestamp‚1.7858043034222913e+09 1loss/FBt1/assignment_loss‚0.14120503021645006 .loss/FBt1/detection_loss‚5.967804533135887 %loss/total_loss‚59.21317059729631 epoch‚0 1loss/SRqqt2/assignment_loss‚2.610905215304385 _step‚91W>%ìú _step‚91 2loss/SRqqt1/assignment_loss‚2.2408299351280387 1loss/FBt2/assignment_loss‚0.41993311177939174 .loss/FRt1/detection_loss‚5.446856138110161 /loss/FRt2/detection_loss‚3.0938392751807466 /loss/SRqqt1/detection_loss‚2.16483876569958 .loss/FBt2/detection_loss‚23.43605588003993 _runtime‚ 492.695475207 /loss/FRt1/assignment_loss‚5.494402162233988 /loss/FRt2/assignment_loss‚4.943970594931086 0loss/SRqqt2/detection_loss‚3.252529955536668 trainer/global_step‚1149 % _timestamp‚1.7858043034222913e+09 1loss/FBt1/assignment_loss‚0.14120503021645006 .loss/FBt1/detection_loss‚5.967804533135887 %loss/total_loss‚59.21317059729631 epoch‚0 1loss/SRqqt2/assignment_loss‚2.610905215304385[}Edœí‚(:‘ ‘ìÄÓëÄ°Û gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚143.141' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚79.52277777777778 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓÈ[J}î‚(:ò ‘ìÄÓÁí ß network.recv‚111298) disk./.usagePercent‚82.42652000070126 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚161.5 disk.sda.out‚ 23.2890625 disk.rbd4.out‚ 0.06298828125 disk.rbd5.in‚0 disk.sda1.out‚0 network.sent‚ 1.036068e+06' disk.nvme0c0n1.out‚2347.71728515625 disk.rbd1.in‚0 disk.rbd1.out‚ 0.1806640625% memory_percent‚41.208380460739136 proc.cpu.threads‚29 disk.sda3.out‚ 23.2890625" disk.dm-0.out‚2347.71728515625 disk.rbd5.out‚ 0.0859375 disk.drbd1743.in‚0 disk.sda2.out‚0 disk.sda.in‚0( proc.memory.percent‚17.1908438205719$ disk./.usageGB‚1228.214714050293 disk.drbd1743.out‚ 0.5078125 disk.rbd3.out‚0 disk.nvme0c0n1.in‚161.5 disk.rbd0.in‚0 disk.rbd2.out‚ 0.1806640625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 23.2890625 disk.nvme0n1.in‚161.5 disk.sda3.in‚0 cpu‚14.371014715470217% disk.nvme0n1.out‚2347.71728515625 disk.rbd0.out‚0 disk.rbd4.in‚0* proc.memory.availableMB‚38529.67578125$ proc.memory.rssMB‚11266.19140625 DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚21145774304568 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12668697 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚21660144852480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12668697 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20644874 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚21660249321472 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1055591031808 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4726797 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4726795 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚21738845161984 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12678353 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1052920000612 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚20604641250304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3445933 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20092751461472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚21145671462084 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20092768500832 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚21660144852480 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12678353 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3446188 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21738683292672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1084318797824 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 _runtime‚ 512.497552507 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1055503602176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20654532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8172728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7976179 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚1199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1084455763456 % _timestamp‚1.7858043232203517e+09 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7976177 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7976177 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20604641250304 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8172985 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 _step‚93 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚21145774304568 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12668697 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚21660144852480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12668697 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20644874 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚21660249321472 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1055591031808 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4726797 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4726795 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚21738845161984 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12678353 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1052920000612 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚20604641250304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3445933 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20092751461472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚21145671462084 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20092768500832 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚21660144852480 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12678353 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3446188 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21738683292672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1084318797824 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 _runtime‚ 512.497552507 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1055503602176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20654532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8172728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7976179 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚1199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1084455763456 % _timestamp‚1.7858043232203517e+09 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7976177 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7976177 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20604641250304 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8172985 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 _step‚93 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3448776 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21756955401216 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20671712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12688890 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1056383221760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1085358885888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1085215635968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _step‚94 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚21678295711232 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚20621825052672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1056470658560 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20662040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 trainer/global_step‚1199 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚20621808013312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1056383221760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20671712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20671571611648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3449036 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 512.837023099 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 143249920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7982822 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8179508 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚21163289107626 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20109508673152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392s0¶À&ùºM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4730732 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12679220 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20662040 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7982822 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1056383221760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1085358885888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1085215635968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _step‚94 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚21678295711232 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚20621825052672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1056470658560 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20662040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 trainer/global_step‚1199 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚20621808013312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1056383221760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20671712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20671571611648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3449036 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 512.837023099 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 143249920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7982822 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8179508 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚21163289107626 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20109508673152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12679220 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚20621808013312 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4730734 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1056470658560 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8179770 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚20671596515328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚21678295711232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1053883277070 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚21163391950222 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3448776 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21756955401216 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20671712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12688890 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13206355 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8308327 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22562479138304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22026671384076 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8308329 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8512918 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4923645 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3589015 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22562583607296 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22562479138304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚22644329886720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1099499737600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 trainer/global_step‚1249 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚21462996440064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1099499737600 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3589271 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13206355 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚21462979400704 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚21462979400704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251› ÐB¶'„°O WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8512660 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22562583607296 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22026568541592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1129451876864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21514684 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4923647 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1129584648192 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20929760077472 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _step‚97 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8308327 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1096894267244 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 % _timestamp‚1.7858043449709752e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13206355 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚21462979400704 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚21462979400704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13206355 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8308327 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22562479138304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22026671384076 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8308329 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8512918 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4923645 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3589015 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22562583607296 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22562479138304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚22644329886720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1099499737600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 trainer/global_step‚1249 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚21462996440064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1099499737600 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3589271 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13216892 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8314972 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚21480146163712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3592095 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1130486181376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 % _timestamp‚1.7858043453738263e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8519427 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3591845 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 534.647665395 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21531864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚21531952355328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1130355514368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22580525520896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8314970 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22580629997056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚21480163203072 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13216892 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22044186187134 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22662282966016 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13207222 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚1249 _step‚98 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8314970 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚20946500249792 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚1249 _step‚98 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8314970 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚20946500249792 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13216892 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8314972 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚21480146163712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3592095 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1130486181376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 % _timestamp‚1.7858043453738263e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8519427 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3591845 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 534.647665395 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21531864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚21531952355328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1130355514368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22580525520896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8314970 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22580629997056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚21480163203072 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13216892 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22044186187134 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22662282966016 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13207222 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚23549747226624 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22321334590464 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22321317551104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚22375076174848 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8852003 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚21766785732832 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13724701 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1140782730752 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚21766768693472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1143495873024 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8640479 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚23464813424128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1143583302656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1143583302656 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚23464917893120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1174671051776 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13734357 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8851760 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22321334590464 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23549902804480 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5120495 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22365178 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22321317551104 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1140696927628 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13724701 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5120497 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3731265 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22375101078528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13734357 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8640477 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8640477 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8640479 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚23464813424128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 _step‚101 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚23464917893120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1174801725952 _runtime‚ 554.495555617 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22374836 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚23549747226624 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22321334590464 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22321317551104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚22375076174848 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8852003 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚21766785732832 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13724701 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1140782730752 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚21766768693472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1143495873024 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8640479 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚23464813424128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1143583302656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1143583302656 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚23464917893120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1174671051776 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13734357 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8851760 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22321334590464 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23549902804480 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5120495 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22365178 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22321317551104 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1140696927628 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13724701 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5120497 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3731265 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22375101078528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13734357 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8640477 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8640477 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8640479 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚23464813424128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 _step‚101 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚23464917893120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2e—"/0“+ _wandbruntime‚554  _runtime‚554þîÞ¼&”¶M S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚23482964282880 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚22392283291648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚1299 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1144375492608 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5124432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚23482964282880 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3734131 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22925186109238 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚21783508865792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22338501353472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1175573566976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22338484314112 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22382344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7858043655607102e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22392016 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13735224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1144462929408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8858563 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13744894 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8647120 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22392308195328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1141660204086 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚23482859806720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22382344 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5124434 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _runtime‚ 554.83429789 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8858817 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚23567856858624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13744894 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1175706331136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚23568014526464 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22392016 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8647120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032l°¨×v•ºM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22338501353472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1175573566976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_¬úsJ$bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22338484314112 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22382344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7858043655607102e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22392016 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13735224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1144462929408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8858563 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13744894 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8647120 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22392308195328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1141660204086 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚23482859806720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22382344 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5124434 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _runtime‚ 554.83429789 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8858817 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚23567856858624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13744894 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1175706331136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚23568014526464 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22392016 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8647120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚23482964282880 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚22392283291648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚1299 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1144375492608 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5124432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚23482964282880 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3734131 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22925186109238 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚21783508865792f'>Î80–+ _wandbruntime‚554  _runtime‚554‘, —ú .loss/FRt2/detection_loss‚4.179653507507878 .loss/FBt1/detection_loss‚5.794178576946714 _step‚103 /loss/FRt1/assignment_loss‚4.831886617148795 /loss/FBt2/detection_loss‚16.922875725431368 epoch‚0 _runtime‚ 554.835091489 2loss/SRqqt1/assignment_loss‚2.0154032465070486 2loss/SRqqt2/assignment_loss‚2.5323790416382908 /loss/FBt2/assignment_loss‚0.609377475362271 1loss/SRqqt1/detection_loss‚2.3241585234607114 trainer/global_step‚1299 /loss/FRt2/assignment_loss‚4.509676712530631 0loss/SRqqt2/detection_loss‚2.425882697591748 &loss/total_loss‚50.375991579485316 % _timestamp‚1.7858043655619895e+09 1loss/FBt1/assignment_loss‚0.23676921861312752 /loss/FRt1/detection_loss‚3.9937502367467417s‡È˜þ 1loss/FBt1/assignment_loss‚0.23676921861312752 /loss/FRt1/detection_loss‚3.9937502367467417 .loss/FRt2/detection_loss‚4.179653507507878 .loss/FBt1/detection_loss‚5.794178576946714 _step‚103 /loss/FRt1/assignment_loss‚4.831886617148795 /loss/FBt2/detection_loss‚16.922875725431368 epoch‚0 _runtime‚ 554.835091489 2loss/SRqqt1/assignment_loss‚2.0154032465070486 2loss/SRqqt2/assignment_loss‚2.5323790416382908 /loss/FBt2/assignment_loss‚0.609377475362271 1loss/SRqqt1/detection_loss‚2.3241585234607114 trainer/global_step‚1299 /loss/FRt2/assignment_loss‚4.509676712530631 0loss/SRqqt2/detection_loss‚2.425882697591748 &loss/total_loss‚50.375991579485316 % _timestamp‚1.7858043655619895e+09gÔ6☛™‚(: ÍìÄÓ§ÞÛ gpu.0.gpu‚15.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚119.86' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚66.58888888888889 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦>v}š‚(:ô ÍìÄÓ–Äà% disk./.usageGB‚1228.2155456542969 disk.dm-0.in‚161.5 disk.rbd4.in‚0 disk.dm-1.out‚ 26.0859375 disk.nvme0c0n1.in‚161.5$ proc.memory.rssMB‚11266.19140625 proc.cpu.threads‚29 disk.rbd2.out‚ 0.306640625$ memory_percent‚41.21151566505432* proc.memory.availableMB‚38527.62109375 disk.dm-1.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.drbd1743.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 network.recv‚120748 disk.sda3.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 26.0859375 disk.rbd4.out‚ 0.21142578125 disk.sda1.out‚0 disk.rbd1.out‚ 0.306640625 cpu‚14.337750159886378 network.sent‚ 1.173867e+06" disk.dm-0.out‚2411.14599609375 disk.rbd5.out‚ 0.1806640625 disk.rbd2.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚161.5% disk.nvme0n1.out‚2411.14599609375) disk./.usagePercent‚82.42657581034372 disk.sda3.out‚ 26.0859375 disk.sda1.in‚0' disk.nvme0c0n1.out‚2411.14599609375 disk.sda.in‚0 disk.rbd5.in‚0 disk.drbd1743.out‚ 0.57421875 disk.rbd3.in‚0( proc.memory.percent‚17.1908438205719³ øœ›‚(:‘ ÜìÄÓƒá¿Û gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚153.512' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚85.28444444444445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œK'¶{œ‚(:ð ÜìÄÓ³ÿ´ß disk.sda1.out‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 38528.2890625) disk./.usagePercent‚82.36467395658703 disk.rbd1.out‚ 0.306640625 disk.sda3.out‚ 26.734375 disk.rbd4.in‚0 disk.rbd4.out‚ 0.24365234375$ memory_percent‚41.21049642562866 proc.cpu.threads‚29 disk.sda.in‚0 disk.rbd2.out‚ 0.306640625 disk.rbd1.in‚0 disk.nvme0n1.in‚161.5 disk.rbd5.out‚ 0.24365234375$ proc.memory.rssMB‚11266.19140625 cpu‚14.332139102262209 disk.nvme0c0n1.in‚161.5% disk.nvme0n1.out‚2473.63427734375 disk.sda.out‚ 26.734375 disk.rbd3.in‚0 disk.sda2.in‚0 disk.drbd1743.out‚ 0.5859375 disk.rbd3.out‚0" disk.dm-0.out‚2473.63427734375 disk.dm-1.out‚ 26.734375 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚2473.63427734375 network.sent‚ 1.218826e+06 network.recv‚123213 disk.rbd2.in‚0 disk.sda3.in‚0( proc.memory.percent‚17.1908438205719% disk./.usageGB‚1227.2931632995605 disk.dm-0.in‚161.5 disk.rbd5.in‚0 disk.dm-1.in‚0 disk.drbd1743.in‚0 disk.sda1.in‚0,`—†0+ _wandbruntime‚574  _runtime‚574ABŒÜÆžÀ & lr-AdamW/pg1‚3.476667956867793e-05 & lr-AdamW/pg2‚3.476667956867793e-05 trainer/global_step‚1349 % _timestamp‚1.7858043850254514e+09 _runtime‚ 574.298743928 _step‚104ê-ÊŸÄ & lr-AdamW/pg1‚3.476667956867793e-05 & lr-AdamW/pg2‚3.476667956867793e-05 trainer/global_step‚1349 % _timestamp‚1.7858043850254514e+09 _runtime‚ 574.298743928 _step‚104h ì?0 + _wandbruntime‚574  _runtime‚5740ïm³'¡­O L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3874199 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1184671194260 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚23179655701504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9191544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 _step‚105 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 trainer/global_step‚1349 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1219849913344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14262359 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23235432014848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1219976393216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚23788465543092 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1187492008448 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚23179655701504 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24367252178944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚23179672740864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22603777309472 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚24367252178944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1184585391136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5317345 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1187579438080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8972627 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1187492008448 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23235456918528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1187579438080 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24455433311744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14252703 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24367147709952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14252703 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9191799 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _runtime‚ 574.303559349 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14262359 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22603794348832 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8972629 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8972629 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5317347 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1219849913344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14262359 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23235432014848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1219976393216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚23788465543092 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1187492008448 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚23179655701504 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24367252178944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚23179672740864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22603777309472 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚24367252178944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1184585391136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5317345 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1187579438080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8972627 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1187492008448 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23235456918528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1187579438080 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24455433311744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14252703 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24367147709952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14252703 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9191799 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _runtime‚ 574.303559349 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14262359 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22603794348832 YKDevicØá²keStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8972629 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8972629 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5317347 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 574.637649868 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9198248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14263226 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5321284 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23252168 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5321282 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1220748979712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚23196839503872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14263226 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚23196822464512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 % _timestamp‚1.7858043853641217e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8979270 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3876966 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1220885938176 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887Ž» ïÁ&¥»M QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23252168 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5321282 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1220748979712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚23196839503872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14263226 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚23196822464512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 % _timestamp‚1.7858043853641217e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8979270 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3876966 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1220885938176 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14272896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 trainer/global_step‚1349 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 574.637649868 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9198248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14263226 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5321284 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0jÑX£0¦+ _wandbruntime‚574  _runtime‚574én~§ú /loss/FRt1/assignment_loss‚4.596684778437895 1loss/SRqqt1/detection_loss‚1.9380271740589077 /loss/FBt2/detection_loss‚14.786645475272302 _runtime‚ 574.638755111 _step‚107 1loss/FBt1/assignment_loss‚0.24359441690903622 1loss/FBt2/assignment_loss‚0.49262271114540374 /loss/FBt1/detection_loss‚2.8668173661663188 %loss/total_loss‚44.90361783611681 trainer/global_step‚1349 % _timestamp‚1.7858043853653722e+09 1loss/SRqqt1/assignment_loss‚1.857628900768622 /loss/FRt1/detection_loss‚3.8213581190389747 .loss/FRt2/detection_loss‚4.078412410758791 0loss/SRqqt2/detection_loss‚3.036789860161831 epoch‚0 /loss/FRt2/assignment_loss‚4.601507910660335 1loss/SRqqt2/assignment_loss‚2.583528712738392Ý¥ª¨þ 1loss/SRqqt2/assignment_loss‚2.583528712738392 /loss/FRt1/assignment_loss‚4.596684778437895 1loss/SRqqt1/detection_loss‚1.9380271740589077 /loss/FBt2/detection_loss‚14.786645475272302 _runtime‚ 574.638755111 _step‚107 1loss/FBt1/assignment_loss‚0.24359441690903622 1loss/FBt2/assignment_loss‚0.49262271114540374 /loss/FBt1/detection_loss‚2.8668173661663188 %loss/total_loss‚44.90361783611681 trainer/global_step‚1349 % _timestamp‚1.7858043853653722e+09 1loss/SRqqt1/assignment_loss‚1.857628900768622 /loss/FRt1/detection_loss‚3.8213581190389747 .loss/FRt2/detection_loss‚4.078412410758791 0loss/SRqqt2/detection_loss‚3.036789860161831 epoch‚0 /loss/FRt2/assignment_loss‚4.601507910660335kÈè¡âœ©‚(:‘ ëìÄÓ“À·Û gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚141.781' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚78.76722222222222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛÞ‹Qª‚(:ö ëìÄÓ¢£‘ß disk.drbd1743.in‚0 disk.rbd2.out‚ 0.306640625 disk.nvme0c0n1.in‚161.5% disk.nvme0n1.out‚2681.76416015625( disk./.usagePercent‚82.3782748688586 disk.sda.out‚ 27.48828125 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 0.306640625 disk.nvme0n1.in‚161.5 disk.dm-0.in‚161.5 disk.sda.in‚0( proc.memory.percent‚17.1908438205719 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ proc.memory.rssMB‚11266.19140625 network.recv‚124537 disk.rbd4.out‚ 0.24365234375 disk.sda1.in‚0' disk.nvme0c0n1.out‚2681.76416015625 disk.rbd1.in‚0 disk.drbd1743.out‚ 0.5859375 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 27.48828125 disk.rbd0.in‚0% memory_percent‚41.210877895355225) proc.memory.availableMB‚ 38528.0390625 cpu‚14.326553185121272% disk./.usageGB‚1227.4958267211914" disk.dm-0.out‚2681.76416015625 disk.rbd4.in‚0 disk.rbd5.in‚0 disk.dm-1.out‚ 27.48828125 network.sent‚ 1.221449e+06 disk.rbd5.out‚ 0.24365234375 disk.sda3.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚29 Þ¿0«+ _wandbruntime‚593  _runtime‚593óJÿ“ƬÀ & lr-AdamW/pg1‚3.605528889294323e-05 & lr-AdamW/pg2‚3.605528889294323e-05 trainer/global_step‚1399 % _timestamp‚1.7858044045234165e+09 _runtime‚ 593.796750284 _step‚108§ùÓÊ­Ä % _timestamp‚1.7858044045234165e+09 _runtime‚ 593.796750284 _step‚108 & lr-AdamW/pg1‚3.605528889294323e-05 & lr-AdamW/pg2‚3.605528889294323e-05 trainer/global_step‚1399l½þS 0®+ _wandbruntime‚593  _runtime‚593Ù›°ˆD¯­O WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1231488143872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1264996717568 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14790361 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚25360946441728 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚25269481995776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14780705 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24038010891264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24038010891264 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚23440785925472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1228473854644 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9530796 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14780705 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚23440802964832 _runtime‚ 593.801572348 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24037993851904 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1231488143872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚109 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚25269586464768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24095140 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9530523 WKDeviceStatsMonitor.on_train_batch_startiì: o /inactive_split.large_pool.allocated‚5514197 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5514195 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚25269586464768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚24095787854848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 % _timestamp‚1.7858044045244353e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚24669362622600 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9304779 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚25269481995776 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9304777 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1231575573504 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚25269481995776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14780705 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24038010891264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24038010891264 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚23440785925472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1228473854644 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9530796 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14780705 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚23440802964832 _runtime‚ 593.801572348 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24037993851904 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1231488143872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚109 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚25269586464768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24095140 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9530523 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5514197 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5514195 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚25269586464768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚24095787854848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 % _timestamp‚1.7858044045244353e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚24669362622600 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9304779 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚25269481995776 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9304777 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1231575573504 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24112320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5518134 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1266034757632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9311420 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚24686980268254 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚110 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9537270 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1232367763456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚1399 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14791228 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1232367763456 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9537530 _runtime‚ 594.134012227 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14800898 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚25287632854528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1229437131102 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24112320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4019138 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25378894867968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1265899896320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1232455200256 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14791228 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚23457526097792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚24112994971648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 % _timestamp‚1.7858044048604107e+09 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9311422 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9537270 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1232367763456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚1399 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14791228 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1232367763456 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9537530 _runtime‚ 594.134012227 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14800898 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚25287632854528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1229437131102 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24112320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4019138 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25378894867968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1265899896320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1232455200256 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14791228 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚23457526097792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚24112994971648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 % _timestamp‚1.7858044048604107e+09 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9311422 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24112320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5518134 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670n ô×0´+ _wandbruntime‚594  _runtime‚594YRþµø /loss/FRt1/assignment_loss‚4.472206641268986 /loss/FRt1/detection_loss‚3.5078588397272172 1loss/SRqqt1/detection_loss‚1.9901223587256032 %loss/total_loss‚48.38814567115551 epoch‚0 % _timestamp‚1.7858044048616242e+09 _runtime‚ 594.13500017 _step‚111 /loss/FRt2/assignment_loss‚4.470495428357806 trainer/global_step‚1399 2loss/SRqqt1/assignment_loss‚1.5892081879967943 1loss/SRqqt2/assignment_loss‚2.641055591482865 1loss/FBt1/assignment_loss‚0.11346036693654381 0loss/FBt2/assignment_loss‚0.3559055448031837 0loss/SRqqt2/detection_loss‚2.800729284579278 .loss/FBt1/detection_loss‚5.452154652787049 .loss/FBt2/detection_loss‚16.95045854408741 /loss/FRt2/detection_loss‚4.0444902304027766¸uªj¶ü 2loss/SRqqt1/assignment_loss‚1.5892081879967943 1loss/SRqqt2/assignment_loss‚2.641055591482865 1loss/FBt1/assignment_loss‚0.11346036693654381 0loss/FBt2/assignment_loss‚0.3559055448031837 0loss/SRqqt2/detection_loss‚2.800729284579278 .loss/FBt1/detection_loss‚5.452154652787049 .loss/FBt2/detection_loss‚16.95045854408741 /loss/FRt2/detection_loss‚4.0444902304027766 /loss/FRt1/assignment_loss‚4.472206641268986 /loss/FRt1/detection_loss‚3.5078588397272172 1loss/SRqqt1/detection_loss‚1.9901223587256032 %loss/total_loss‚48.38814567115551 epoch‚0 % _timestamp‚1.7858044048616242e+09 _runtime‚ 594.13500017 _step‚111 /loss/FRt2/assignment_loss‚4.470495428357806 trainer/global_step‚1399orèÐ@·‚(:„ úìÄÓÉóãÛ gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚120.15' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚66.75 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†-|¸‚(:ñ úìÄÓ“‚ß disk.rbd3.in‚0 disk.rbd1.out‚ 0.306640625 disk.rbd4.in‚0 disk.dm-1.out‚ 28.1484375 disk.sda1.in‚0 disk.nvme0n1.in‚161.5 disk.dm-0.in‚161.5 disk.sda.out‚ 28.1484375 network.recv‚126995% disk.nvme0n1.out‚2681.97900390625 disk.rbd0.in‚0" disk.dm-0.out‚2681.97900390625 disk.sda.in‚0 disk.rbd4.out‚ 0.306640625 disk.drbd1743.in‚0$ memory_percent‚41.21094346046448 network.sent‚ 1.266448e+06% disk./.usageGB‚1227.4957695007324 disk.sda3.out‚ 28.1484375 disk.rbd0.out‚0* proc.memory.availableMB‚38527.99609375$ proc.memory.rssMB‚11266.19140625( proc.memory.percent‚17.1908438205719 cpu‚14.320307234175413 disk.rbd2.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚2681.97900390625 disk.rbd5.out‚ 0.306640625 proc.cpu.threads‚29 disk.sda1.out‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚161.5 disk.rbd5.in‚0 disk.drbd1743.out‚ 0.61328125) disk./.usagePercent‚82.37827102874559 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd2.out‚ 0.306640625Ò2Å0¹+ _wandbruntime‚613  _runtime‚613wtcǺÁ _runtime‚ 613.607138363 _step‚112 ' lr-AdamW/pg1‚3.7343898217208536e-05 ' lr-AdamW/pg2‚3.7343898217208536e-05 trainer/global_step‚1449 $ _timestamp‚1.785804424333804e+09ǘˆ”˻Š' lr-AdamW/pg1‚3.7343898217208536e-05 ' lr-AdamW/pg2‚3.7343898217208536e-05 trainer/global_step‚1449 $ _timestamp‚1.785804424333804e+09 _runtime‚ 613.607138363 _step‚112pb!ÞÒ0¼+ _wandbruntime‚613  _runtime‚613hp<²'½¬O SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚26171920750592 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15318363 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9869933 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1275571708928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚24277811580832 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24896349041664 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1275484279296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26171816281600 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1275484279296 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4158888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1275571708928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24955292 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚25550156859624 trainer/global_step‚1449 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4159147 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1310284670464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15318363 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26171816281600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15308707 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24896332002304 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24956168598528 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚24896332002304 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24896349041664 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚24277794541472 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5711045 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24945634 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚26266295593984 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9636927 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1310151899136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚ 613.611573031 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15308707 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5711047é õý¶'¾°O [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24896332002304 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24956168598528 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚24896332002304 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24896349041664 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚24277794541472 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5711045 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24945634 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚26266295593984 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9636927 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1310151899136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚ 613.611573031 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15308707 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5711047 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚26171920750592 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15318363 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9869933 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1275571708928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚24277811580832 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24896349041664 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1275484279296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26171816281600 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1275484279296 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4158888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1275571708928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24955292 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚25550156859624 trainer/global_step‚1449 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4159147 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1310284670464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15318363 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26171816281600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15308707 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640qÍDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5714982 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5714984 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4161915 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7858044246792579e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9643570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1276451335680 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9643570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚26189862664192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚24913498765312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1276363898880 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24962800 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9643572 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚24913498765312 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4161682 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚25567877347762 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚24294551753152 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24972472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚25567774505166 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _runtime‚ 613.952831496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚24294534713792 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1273239791374 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 trainer/global_step‚1449 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚24913515804672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15328900 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15319230 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1276363898880 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26189967140352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 bODeviceStatsMonitor.on_train_batch_end/inactive_split¤'Û_bytes.large_pool.allocated‚24973375715328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24962800 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1276451335680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15319230 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15328900 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9876899 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9876664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1311184475136 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚25567877347762 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚24294551753152 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24972472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚25567774505166 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _runtime‚ 613.952831496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚24294534713792 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1273239791374 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 trainer/global_step‚1449 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚24913515804672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15328900 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15319230 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1276363898880 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26189967140352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚24973375715328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24962800 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1276451335680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15319230 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15328900 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9876899 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9876664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1311184475136 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5714982 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5714984 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4161915 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7858044246792579e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9643570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1276451335680 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9643570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚26189862664192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0r¼kd0Â+ _wandbruntime‚613  _runtime‚613-lÝ üÃö 0loss/FBt2/assignment_loss‚0.4755125396292318 .loss/FBt1/detection_loss‚7.382456125700054 epoch‚0 $ _timestamp‚1.785804424680463e+09 _runtime‚ 613.953572917 _step‚115 /loss/FRt1/assignment_loss‚4.421010186455467 /loss/FRt2/assignment_loss‚4.624569339850514 2loss/SRqqt1/assignment_loss‚1.5515444495811546 0loss/SRqqt1/detection_loss‚1.901145366030872 .loss/FBt2/detection_loss‚29.72184155285223 /loss/FRt2/detection_loss‚3.5064871199813084 0loss/SRqqt2/detection_loss‚3.831692832045955 .loss/FRt1/detection_loss‚5.772376428409056 %loss/total_loss‚65.90643491959068 trainer/global_step‚1449 2loss/SRqqt2/assignment_loss‚2.5621355440043194 0loss/FBt1/assignment_loss‚0.1556634350505192˪UêÄú _step‚115 /loss/FRt1/assignment_loss‚4.421010186455467 /loss/FRt2/assignment_loss‚4.624569339850514 2loss/SRqqt1/assignment_loss‚1.5515444495811546 0loss/SRqqt1/detection_loss‚1.901145366030872 .loss/FBt2/detection_loss‚29.72184155285223 /loss/FRt2/detection_loss‚3.5064871199813084 0loss/SRqqt2/detection_loss‚3.831692832045955 .loss/FRt1/detection_loss‚5.772376428409056 %loss/total_loss‚65.90643491959068 trainer/global_step‚1449 2loss/SRqqt2/assignment_loss‚2.5621355440043194 0loss/FBt1/assignment_loss‚0.1556634350505192 0loss/FBt2/assignment_loss‚0.4755125396292318 .loss/FBt1/detection_loss‚7.382456125700054 epoch‚0 $ _timestamp‚1.785804424680463e+09 _runtime‚ 613.953572917s÷yüi‘Å‚(:† ‰íÄÓüÃÛ gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚144.333' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚80.185 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö¡Ç=xÆ‚(:í ‰íÄÓþ×€Þ disk.sda3.out‚ 28.98828125 disk.drbd1743.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚161.5! disk.dm-0.out‚2682.3955078125) proc.memory.availableMB‚ 38527.7109375$ proc.memory.rssMB‚11266.19140625 disk.rbd1.out‚ 0.306640625 disk.rbd5.in‚0 disk.dm-1.in‚0$ memory_percent‚41.21137857437134 network.sent‚ 1.311719e+06 disk.sda2.out‚0 disk.drbd1743.out‚ 0.6328125 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.rbd4.in‚0 cpu‚14.314193573687255 network.recv‚130226( disk./.usagePercent‚82.3782764049038 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda.out‚ 28.98828125 proc.cpu.threads‚29$ disk./.usageGB‚1227.495849609375 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd4.out‚ 0.306640625 disk.rbd5.out‚ 0.306640625 disk.dm-1.out‚ 28.98828125( proc.memory.percent‚17.1908438205719 disk.sda1.out‚0 disk.nvme0n1.in‚161.5 disk.sda3.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚161.5 disk.rbd2.out‚ 0.306640625 disk.rbd3.in‚0& disk.nvme0c0n1.out‚2682.3955078125$ disk.nvme0n1.out‚2682.3955078125ºÞ¼ÅœÇ‚(:‘ ˜íÄÓËÃ¥Ü gpu.0.gpu‚44.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚151.905' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚84.39166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I:ƒçzÈ‚(:ï ˜íÄÓú…ß disk.sda3.out‚ 29.49609375 disk.rbd2.out‚ 0.306640625 disk.rbd1.in‚0$ proc.memory.rssMB‚11266.20703125 disk.dm-0.out‚2682.689453125 disk.rbd4.in‚0 disk.dm-1.out‚ 29.49609375# memory_percent‚41.2106454372406 cpu‚14.308409223089834% disk./.usageGB‚1227.9978218078613 disk.drbd1743.in‚0 disk.nvme0c0n1.in‚161.5 disk.dm-0.in‚161.5 disk.sda.in‚0 disk.rbd5.out‚ 0.306640625* proc.memory.availableMB‚38528.19140625 disk.sda2.out‚0 disk.nvme0n1.in‚161.5# disk.nvme0n1.out‚2682.689453125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 29.49609375 disk.rbd5.in‚0 network.recv‚132919) disk./.usagePercent‚82.41196418032696 disk.sda1.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda1.out‚0% disk.nvme0c0n1.out‚2682.689453125 disk.sda3.in‚0 disk.rbd4.out‚ 0.36962890625 disk.dm-1.in‚0 disk.rbd3.out‚0 network.sent‚ 1.356724e+06 disk.sda2.in‚0) proc.memory.percent‚17.19086766242981 proc.cpu.threads‚29 disk.drbd1743.out‚ 0.6484375 disk.rbd2.in‚0 disk.rbd3.in‚0yú×z0É+ _wandbruntime‚632  _runtime‚632 îéÆÊÀ % _timestamp‚1.7858044436800098e+09 _runtime‚ 632.953312394 _step‚116 & lr-AdamW/pg1‚3.863250754147384e-05 & lr-AdamW/pg2‚3.863250754147384e-05 trainer/global_step‚14997bÅÊËÄ _runtime‚ 632.953312394 _step‚116 & lr-AdamW/pg1‚3.863250754147384e-05 & lr-AdamW/pg2‚3.863250754147384e-05 trainer/global_step‚1499 % _timestamp‚1.7858044436800098e+09tÉÚ;m0Ì+ _wandbruntime‚632  _runtime‚632[nˆµ'ͯO BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚25114803157472 _step‚117 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10209248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25754687192064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10209502 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 _runtime‚ 632.957612619 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25816524438528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25816499534848 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚27074255036416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚27074255036416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27172070955008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1355546516480 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25114820196832 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1319567844352 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9969077 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9969077 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚27074150567424 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27171923765760 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚26431053939132 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9969079 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15836709 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1319567844352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5907895 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1319480414720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15846365 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4301353 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1316250781660 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27074150567424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4301605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122285568 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1319480414720 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5907897 % _timestamp‚1.7858044436810603e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25805786 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1355424230912 —$F γO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15836709 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1319567844352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5907895 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1319480414720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15846365 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4301353 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1316250781660 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27074150567424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4301605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122285568 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1319480414720 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5907897 % _timestamp‚1.7858044436810603e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25805786 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1355424230912 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚25114803157472 _step‚117 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10209248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25754687192064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10209502 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 _runtime‚ 632.957612619 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25816524438528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25816499534848 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚27074255036416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚27074255036416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27172070955008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1355546516480 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25114820196832 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1319567844352 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9969077 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9969077 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚27074150567424 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27171923765760 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚26431053939132 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9969079 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658u¨wUv0Ï+ _wandbruntime‚633  _runtime‚633¨ Ù÷¿&йM B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27190026647552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26448671584674 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚25771853955072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1320360034304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9975720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25832624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9975722 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10216297 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1317128254882 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚27092301426176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5911832 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1356319995904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚26448774427270 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4304463 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27190190606848 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9975720 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1320360034304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1320447471104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10216030 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _runtime‚ 633.291346251 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚27092301426176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15847232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4304198 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚25833706651648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15856902 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25822952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1356459051520 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1317214058118 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9975722 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15847232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25832624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚25131543329792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 % _timestamp‚1.7858044440178175e+09 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0°óvuÃ&ѽM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚26448774427270 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4304463 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27190190606848 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9975720 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1320360034304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1320447471104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10216030 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _runtime‚ 633.291346251 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚27092301426176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15847232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4304198 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚25833706651648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15856902 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25822952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1356459051520 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1317214058118 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9975722 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15847232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25832624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚25131543329792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 % _timestamp‚1.7858044440178175e+09 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27190026647552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26448671584674 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚25771853955072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1320360034304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9975720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25832624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9975722 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10216297 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1317128254882 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚27092301426176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5911832 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1356319995904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0v§©¶½0Ò+ _wandbruntime‚633  _runtime‚633ÛߧýÓ÷ .loss/FBt1/detection_loss‚5.610000418088845 1loss/FBt2/assignment_loss‚0.35371663317844076 %loss/total_loss‚48.51835906530178 trainer/global_step‚1499 % _timestamp‚1.7858044440189874e+09 _runtime‚ 633.29211382 /loss/FRt1/assignment_loss‚4.332308614408815 .loss/FRt2/assignment_loss‚4.27026935012973 0loss/FBt1/assignment_loss‚0.1655253509442056 .loss/FRt1/detection_loss‚4.142615333780066 /loss/FRt2/detection_loss‚3.3867539181088913 1loss/SRqqt1/detection_loss‚1.6989437392960365 /loss/FBt2/detection_loss‚17.015148346684356 _step‚119 2loss/SRqqt1/assignment_loss‚1.6486706110564144 1loss/SRqqt2/assignment_loss‚2.483930901636051 0loss/SRqqt2/detection_loss‚3.410475847989925 epoch‚0@~èGÔû 1loss/FBt2/assignment_loss‚0.35371663317844076 %loss/total_loss‚48.51835906530178 trainer/global_step‚1499 % _timestamp‚1.7858044440189874e+09 _runtime‚ 633.29211382 /loss/FRt1/assignment_loss‚4.332308614408815 .loss/FRt2/assignment_loss‚4.27026935012973 0loss/FBt1/assignment_loss‚0.1655253509442056 .loss/FRt1/detection_loss‚4.142615333780066 /loss/FRt2/detection_loss‚3.3867539181088913 1loss/SRqqt1/detection_loss‚1.6989437392960365 /loss/FBt2/detection_loss‚17.015148346684356 _step‚119 2loss/SRqqt1/assignment_loss‚1.6486706110564144 1loss/SRqqt2/assignment_loss‚2.483930901636051 0loss/SRqqt2/detection_loss‚3.410475847989925 epoch‚0 .loss/FBt1/detection_loss‚5.610000418088845wÑûupœÕ‚(:‘ §íÄÓá ±Û gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚154.222' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚85.67888888888889 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù»µŒeÖ‚(:Ú §íÄÓ€ÿõÞ disk.rbd1.in‚0% disk./.usageGB‚1227.9958686828613 disk.dm-0.in‚161.5 disk.drbd1743.out‚ 0.6484375 disk.sda2.out‚0& proc.memory.availableMB‚ 38528.1875$ proc.memory.rssMB‚11266.20703125 disk.rbd5.out‚ 0.306640625 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0# disk.nvme0n1.out‚2982.236328125 disk.sda3.in‚0 cpu‚14.303745323384366 disk.sda.in‚0 disk.rbd4.out‚ 0.36962890625 disk.dm-1.out‚30.25 disk.drbd1743.in‚0 disk.rbd0.in‚0 network.recv‚134253) disk./.usagePercent‚82.41183310446942 disk.dm-0.out‚2982.236328125 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚161.5 network.sent‚ 1.359184e+06 disk.sda.out‚30.25 disk.nvme0c0n1.in‚161.5% disk.nvme0c0n1.out‚2982.236328125 disk.rbd1.out‚ 0.306640625 disk.sda3.out‚30.25 disk.rbd0.out‚0$ memory_percent‚41.21065139770508 disk.rbd4.in‚0 disk.dm-1.in‚0 disk.rbd2.out‚ 0.306640625 disk.sda1.in‚0) proc.memory.percent‚17.19086766242981 proc.cpu.threads‚29 disk.rbd5.in‚0ŠÆØ0×+ _wandbruntime‚652  _runtime‚652_ ÂŒÈØ ' lr-AdamW/pg2‚3.9921116865739145e-05 trainer/global_step‚1549 % _timestamp‚1.7858044634772918e+09 _runtime‚ 652.752581795 _step‚120 ' lr-AdamW/pg1‚3.9921116865739145e-05ÛBtÌÙÆ _runtime‚ 652.752581795 _step‚120 ' lr-AdamW/pg1‚3.9921116865739145e-05 ' lr-AdamW/pg2‚3.9921116865739145e-05 trainer/global_step‚1549 % _timestamp‚1.7858044634772918e+09xt20Ú+ _wandbruntime‚652  _runtime‚652S¤7¿Û±O IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10549504 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚121 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _runtime‚ 652.7526967 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10301229 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 BDeviceÑ:š®œStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚26613008303104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16374367 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26676880278528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28077597832704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1400717554176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16374367 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚26613025342464 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26675596 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚27976589322240 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚26613008303104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10549758 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚26613025342464 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16364711 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1363563979776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10301227 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6104745 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26676855374848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1363476550144 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27976484853248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1363563979776 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4445011 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28077438060544 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25951828812832 % _timestamp‚1.7858044634779987e+09 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538ãø`a»'ܵO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10301229 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚26613008303104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16374367 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26676880278528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28077597832704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1400717554176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16374367 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚26613025342464 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26675596 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚27976589322240 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚26613008303104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10549758 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚26613025342464 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16364711 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1363563979776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10301227 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6104745 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26676855374848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1363476550144 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27976484853248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1363563979776 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4445011 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28077438060544 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25951828812832 % _timestamp‚1.7858044634779987e+09 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10549504 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚121 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _runtime‚ 652.7526967 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640y$òûW0Ý+ _wandbruntime‚653  _runtime‚653‘[FPÃ&Þ½M SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1364443606528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26694087395328 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26630192105472 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚26630192105472 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚26630175066112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚27994531235840 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28095706073088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚26630175066112 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16384904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26692776 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1361016718390 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1364356169728 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1401618677760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10307872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4447525 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10307870 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚25968551945792 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26683104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10556207 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1364443606528 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27329568664182 % _timestamp‚1.7858044638138738e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚27994635712000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10307872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26692776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16375234 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6108684 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27329671506778 _runtime‚ 653.087575515 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1364356169728 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _step‚122 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26683104 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6108682 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4447793 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10556477 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1361102521626 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1401483816448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0î8•ï“ßÁM ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1401618677760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10307872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4447525 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10307870 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚25968551945792 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26683104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10556207 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1364443606528 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27329568664182 % _timestamp‚1.7858044638138738e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚27994635712000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10307872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26692776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16375234 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6108684 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27329671506778 _runtime‚ 653.087575515 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1364356169728 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _step‚122 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26683104 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6108682 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4447793 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10556477 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1361102521626 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_poR³n:4 ol.freed‚ 1401483816448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1364443606528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26694087395328 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26630192105472 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚26630192105472 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚26630175066112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚27994531235840 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28095706073088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚26630175066112 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16384904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26692776 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1361016718390 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1364356169728 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251z~SÞ0à+ _wandbruntime‚653  _runtime‚653z=íþáø /loss/FRt2/assignment_loss‚4.256187376768692 1loss/FBt1/assignment_loss‚0.12644587150134612 0loss/FBt2/assignment_loss‚0.3529877889552154 0loss/SRqqt1/detection_loss‚2.709655114845911 trainer/global_step‚1549 /loss/FRt1/assignment_loss‚4.570629800519636 1loss/SRqqt1/assignment_loss‚1.530770941414928 1loss/SRqqt2/detection_loss‚3.9236162394820737 epoch‚0 $ _timestamp‚1.785804463815107e+09 _step‚123 2loss/SRqqt2/assignment_loss‚2.6102500542214044 /loss/FBt2/detection_loss‚15.388126113478148 _runtime‚ 653.088225294 .loss/FRt1/detection_loss‚5.124964727990089 -loss/FRt2/detection_loss‚3.54786619965149 /loss/FBt1/detection_loss‚6.5110122207575865 &loss/total_loss‚50.652512449586524¹€¿~âü 1loss/SRqqt2/detection_loss‚3.9236162394820737 epoch‚0 $ _timestamp‚1.785804463815107e+09 _step‚123 2loss/SRqqt2/assignment_loss‚2.6102500542214044 /loss/FBt2/detection_loss‚15.388126113478148 _runtime‚ 653.088225294 .loss/FRt1/detection_loss‚5.124964727990089 -loss/FRt2/detection_loss‚3.54786619965149 /loss/FBt1/detection_loss‚6.5110122207575865 &loss/total_loss‚50.652512449586524 /loss/FRt2/assignment_loss‚4.256187376768692 1loss/FBt1/assignment_loss‚0.12644587150134612 0loss/FBt2/assignment_loss‚0.3529877889552154 0loss/SRqqt1/detection_loss‚2.709655114845911 trainer/global_step‚1549 /loss/FRt1/assignment_loss‚4.570629800519636 1loss/SRqqt1/assignment_loss‚1.530770941414928{‰9s<œã‚(:‘ ¶íÄÓ­ìÌÛ gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚152.443' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚84.69055555555556 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýÓxë{ä‚(:ð ¶íÄÓ£¦šß disk.nvme0n1.in‚161.5# disk.nvme0n1.out‚2982.486328125 disk.rbd2.out‚ 0.306640625 disk.rbd0.out‚0) proc.memory.percent‚17.19086766242981$ proc.memory.rssMB‚11266.20703125 proc.cpu.threads‚29) disk./.usagePercent‚82.41182696028861 disk.sda.in‚0 disk.rbd4.out‚ 0.36962890625 disk.sda2.out‚0 disk.rbd3.out‚0 network.recv‚136218$ disk./.usageGB‚1227.995777130127 disk.dm-1.out‚ 30.91015625 disk.sda1.in‚0 disk.rbd3.in‚0 cpu‚14.298587334712444 disk.rbd4.in‚0 disk.drbd1743.in‚0 disk.sda3.out‚ 30.91015625$ memory_percent‚41.20844006538391* proc.memory.availableMB‚38529.63671875 disk.sda.out‚ 30.91015625 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚161.5 disk.dm-0.out‚2982.486328125 disk.rbd5.in‚0 disk.dm-1.in‚0 disk.rbd2.in‚0% disk.nvme0c0n1.out‚2982.486328125 disk.rbd1.in‚0 disk.sda3.in‚0 network.sent‚ 1.404014e+06 disk.rbd5.out‚ 0.306640625 disk.drbd1743.out‚ 0.69140625 disk.sda1.out‚0 disk.nvme0c0n1.in‚161.5 disk.rbd1.out‚ 0.306640625qú¢‘0å+ _wandbruntime‚672  _runtime‚6722ÃÆæÀ & lr-AdamW/pg1‚4.120972619000445e-05 & lr-AdamW/pg2‚4.120972619000445e-05 trainer/global_step‚1599 % _timestamp‚1.7858044830773704e+09 _runtime‚ 672.350651774 _step‚124ÝI»&ÊçÄ & lr-AdamW/pg2‚4.120972619000445e-05 trainer/global_step‚1599 % _timestamp‚1.7858044830773704e+09 _runtime‚ 672.350651774 _step‚124 & lr-AdamW/pg1‚4.120972619000445e-05|kd{0è+ _wandbruntime‚672  _runtime‚672ÛM€¹'é³O SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4588407 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28983143298048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚28192950940632 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1445907179520 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10890246 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27471346453504 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4588649 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6301595 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚28192848098148 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚26788820389472 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28878819139072 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27471363492864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27535748 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27526090 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1407560115200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1445774408192 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28878923608064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28878819139072 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27537211214848 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10890002 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16892713 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28878923608064 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27471346453504 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1407560115200 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1407472685568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚26788837428832 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10633377 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16902369 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28982985623040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1404113511800 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10633379 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10633379 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16892713 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27537236118528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6301597 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16902369 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27471363492864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360‚ÛM£½'ê·O IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28878819139072 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27537211214848 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10890002 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16892713 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28878923608064 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27471346453504 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1407560115200 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1407472685568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚26788837428832 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10633377 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16902369 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28982985623040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1404113511800 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10633379 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10633379 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16892713 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27537236118528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6301597 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16902369 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27471363492864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4588407 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28983143298048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚28192950940632 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1445907179520 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10890246 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27471346453504 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4588649 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6301595 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚28192848098148 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚26788820389472 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28878819139072 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27471363492864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27535748 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27526090 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1407560115200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1445774408192 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28878923608064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640}ÄïÀ0ë+ _wandbruntime‚672  _runtime‚672”ÛÝÈì½M UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1408352305152 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚27488513216512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16912906 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16912906 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4591260 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 672.687848333 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27552928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10897038 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1404905181898 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27552928 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27488530255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29001253144064 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚27488513216512 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28896969997824 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10640020 trainer/global_step‚1599 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 % _timestamp‚1.7858044834142895e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29001097573376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27543256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6305532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 UHDeviceStatsMonitorã¥IŒ6.on_train_batch_end/requested_bytes.small_pool.current‚85803236 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚28210465743690 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16903236 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1446679241728 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1408352305152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4591504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27554418331648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1446809908736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16903236 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10640022 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1404990985134 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10640022 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10896792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27554443235328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _step‚126 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28210568586286 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚26805560561792 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28896969997824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10640020 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 672.687848333 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27552928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10897038 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1404905181898 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27552928 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27488530255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29001253144064 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚27488513216512 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28896969997824 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10640020 trainer/global_step‚1599 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 % _timestamp‚1.7858044834142895e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29001097573376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27543256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6305532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚28210465743690 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16903236 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1446679241728 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1408352305152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4591504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27554418331648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1446809908736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16903236 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10640022 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1404990985134 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10640022 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10896792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27554443235328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _step‚126 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28210568586286 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚26805560561792 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28896969997824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10640020 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚29888492700672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1451556250624 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17420715 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28386242 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚29781257893888 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1451556250624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1491052125696 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚29781257893888 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚28329701643264 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚29888644084224 % _timestamp‚1.7858045026435106e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11229996 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1451468820992 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4731804 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29073848020140 _step‚129 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚29781153424896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10965529 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6498447 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚28329684603904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10965527 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17420715 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10965527 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28397591958528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27625846044832 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27625829005472 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17430371 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4731551 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚28329701643264 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28329684603904 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1447916172184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚29781153424896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28397567054848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29073745177656 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10965529 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17430371 _runtime‚ 691.920599366 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 XEDeviceStatsMonitor.on_train_batch_ Ðom%start/active_bytes.large_pool.freed‚28329684603904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10965527 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17420715 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10965527 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28397591958528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27625846044832 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27625829005472 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17430371 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4731551 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚28329701643264 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28329684603904 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1447916172184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚29781153424896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28397567054848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29073745177656 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10965529 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17430371 _runtime‚ 691.920599366 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚29888492700672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1451556250624 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17420715 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1308_±f0û+ _wandbruntime‚692  _runtime‚692ËrŸòÃ&ü½M TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6502384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28414774171648 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚28346868406272 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28346868406272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1452348440576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28346851366912 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28346851366912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11236951 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10972170 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10972170 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1452435877376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6502382 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4734567 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1491825031680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _step‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚29799199807488 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚29799304283648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29906599203328 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28403408 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28403408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1452348440576 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17440908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚29091465665794 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1452435877376 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29906761065472 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚29799304283648 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10972172 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 trainer/global_step‚1649 _runtime‚ 692.251561491 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17440908 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17431238 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28413080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10972172 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4734332 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27642586217152 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27642569177792 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1448793645406 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29091362823198 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1448879448642 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28413080 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7858045029779115e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1491961990144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11236714 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¾8ºŠÈ&ýÂM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7858045029779115e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1491961990144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11236714 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6502384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28414774171648 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚28346868406272 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28346868406272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1452348440576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28346851366912 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28346851366912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11236951 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10972170 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10972170 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1452435877376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6502382 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4734567 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1491825031680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _step‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚29799199807488 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚29799304283648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29906599203328 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28403408 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28403408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1452348440576 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17440908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚29091465665794 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1452435877376 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29906761065472 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚29799304283648 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10972172 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 trainer/global_step‚1649 _runtime‚ 692.251561491 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17440908 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17431238 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28413080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10972172 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4734332 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27642586217152 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27642569177792 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1448793645406 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29091362823198 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1448879448642 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28413080 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160‚ˆ]q0þ+ _wandbruntime‚692  _runtime‚692þj8tþÿø 2loss/SRqqt1/assignment_loss‚1.4482527041411775 1loss/FBt1/assignment_loss‚0.16715047871737085 .loss/FRt2/detection_loss‚3.915165897952505 0loss/SRqqt1/detection_loss‚1.692102733333302 1loss/SRqqt2/detection_loss‚2.6245617663638185 trainer/global_step‚1649 _step‚131 /loss/FRt2/assignment_loss‚4.178428000714406 .loss/FBt1/detection_loss‚5.604243147807104 %loss/total_loss‚43.08718073059181 _runtime‚ 692.252492376 /loss/FBt2/detection_loss‚12.902893714894958 0loss/FRt1/assignment_loss‚4.1860764898279665 1loss/SRqqt2/assignment_loss‚2.526805061361064 0loss/FBt2/assignment_loss‚0.4340040934371895 -loss/FRt1/detection_loss‚3.40749664204095 epoch‚0 % _timestamp‚1.7858045029790502e+09G»"t€ý 2loss/SRqqt1/assignment_loss‚1.4482527041411775 1loss/FBt1/assignment_loss‚0.16715047871737085 .loss/FRt2/detection_loss‚3.915165897952505 0loss/SRqqt1/detection_loss‚1.692102733333302 1loss/SRqqt2/detection_loss‚2.6245617663638185 trainer/global_step‚1649 _step‚131 /loss/FRt2/assignment_loss‚4.178428000714406 .loss/FBt1/detection_loss‚5.604243147807104 %loss/total_loss‚43.08718073059181 _runtime‚ 692.252492376 /loss/FBt2/detection_loss‚12.902893714894958 0loss/FRt1/assignment_loss‚4.1860764898279665 1loss/SRqqt2/assignment_loss‚2.526805061361064 0loss/FBt2/assignment_loss‚0.4340040934371895 -loss/FRt1/detection_loss‚3.40749664204095 epoch‚0 % _timestamp‚1.7858045029790502e+09ƒm=£lœ‚(:‘ ãíÄÓÝìÑÜ gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚129.091' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚71.71722222222223 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@#`~‚‚(:ó ãíÄÓß•ˆá disk.rbd1.out‚ 0.306640625 disk.sda3.out‚ 33.1171875 disk.dm-0.in‚161.5 disk.sda.in‚0 disk.nvme0c0n1.in‚161.5 network.recv‚141823 disk.dm-1.out‚ 33.1171875 disk.rbd3.out‚0% disk./.usageGB‚1227.9957008361816 disk.sda3.in‚0 disk.rbd2.out‚ 0.306640625 disk.sda1.out‚0$ memory_percent‚41.20787978172302 network.sent‚ 1.495275e+06) disk./.usagePercent‚82.41182184013792 disk.sda.out‚ 33.1171875 disk.rbd4.in‚0' disk.nvme0c0n1.out‚2983.35400390625 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚161.5$ proc.memory.rssMB‚11266.20703125 disk.rbd5.out‚ 0.306640625 disk.drbd1743.in‚0 disk.drbd1743.out‚ 0.7265625 cpu‚14.285310723450621" disk.dm-0.out‚2983.35400390625% disk.nvme0n1.out‚2983.35400390625* proc.memory.availableMB‚38530.00390625) proc.memory.percent‚17.19086766242981 proc.cpu.threads‚29 disk.rbd4.out‚ 0.36962890625 disk.rbd5.in‚0 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0L™| 0ƒ+ _wandbruntime‚711  _runtime‚711ºtŒÆ„À trainer/global_step‚1699 % _timestamp‚1.7858045226286752e+09 _runtime‚ 711.901965388 _step‚132 & lr-AdamW/pg1‚4.378694483853506e-05 & lr-AdamW/pg2‚4.378694483853506e-05Dh›WË…Å & lr-AdamW/pg1‚4.378694483853506e-05 & lr-AdamW/pg2‚4.378694483853506e-05 trainer/global_step‚1699 % _timestamp‚1.7858045226286752e+09 _runtime‚ 711.901965388 _step‚132„ü¹0†+ _wandbruntime‚711  _runtime‚711ï…[\‚‡³O G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 % _timestamp‚1.7858045226296496e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1495552386048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11569144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6695295 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1495464956416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚30794193125376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17958373 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 trainer/global_step‚1699 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 c%‡`7QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1536245326848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30683592179712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1491804635692 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17958373 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚30794037547520 _step‚133 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17948717 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30683592179712 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30683487710720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1495552386048 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚29188022754304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11569380 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚28462837621472 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1491890438816 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11297679 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚28462854660832 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1536114652672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11297677 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29246394 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29256052 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11297679 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29954745099648 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6695297 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚29257947798528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1495464956416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29954642257164A¸r¾'ˆ¸O IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11297679 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚28462854660832 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1536114652672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11297677 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29246394 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29256052 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11297679 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29954745099648 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6695297 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚29257947798528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1495464956416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29954642257164 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 % _timestamp‚1.7858045226296496e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1495552386048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11569144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6695295 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1495464956416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚30794193125376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17958373 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 trainer/global_step‚1699 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1536245326848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30683592179712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1491804635692 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17958373 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚30794037547520 _step‚133 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17948717 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30683592179712 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30683487710720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1495552386048 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚29188022754304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11569380 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚28462837621472 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1491890438816…?û0‰+ _wandbruntime‚712  _runtime‚712Ðý’Ã&Š½M REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11304320 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29972259902706 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29273232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1537163301376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6699234 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6699232 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚29275130011648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30701638569472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1496432012800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1496344576000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29205189517312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1496344576000 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29263560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚29275154915328 trainer/global_step‚1699 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1492767912150 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚30812318216704 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4876951 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚28479577793792 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1492682108914 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11304322 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1496432012800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17959240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚29972362745302 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30701534093312 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29273232 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17968910 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚30812158451712 % _timestamp‚1.7858045229666848e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11575939玀[õ‹ÂM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1496432012800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17959240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚29972362745302 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30701534093312 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29273232 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17968910 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚30812158451712 % _timestamp‚1.7858045229666848e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11575939 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11304320 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29972259902706 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29273232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1537163301376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6699234 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6699232 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚29275130011648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30701638569472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1496432012800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1496344576000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29205189517312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1496344576000 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29263560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚29275154915328 trainer/global_step‚1699 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1492767912150 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚30812318216704 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4876951 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚28479577793792 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1492682108914 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11304322 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121†/0Œ+ _wandbruntime‚712  _runtime‚7124þU¯ú /loss/FRt1/detection_loss‚4.4824814592088975 .loss/FRt2/detection_loss‚3.992944685242525 /loss/FBt2/detection_loss‚13.348643473630073 &loss/total_loss‚49.677643330484464 1loss/SRqqt1/assignment_loss‚1.386177612569674 1loss/SRqqt2/assignment_loss‚2.394697077862628 epoch‚0 _step‚135 1loss/SRqqt1/detection_loss‚1.4756580410076028 .loss/FBt1/detection_loss‚9.689233336721442 % _timestamp‚1.7858045229678938e+09 _runtime‚ 712.241298132 1loss/SRqqt2/detection_loss‚3.9655607631139436 trainer/global_step‚1699 /loss/FRt1/assignment_loss‚4.203102919033595 /loss/FRt2/assignment_loss‚4.244699771811322 1loss/FBt1/assignment_loss‚0.09769661043343755 0loss/FBt2/assignment_loss‚0.3967475798493251»ßquŽÿ % _timestamp‚1.7858045229678938e+09 _runtime‚ 712.241298132 1loss/SRqqt2/detection_loss‚3.9655607631139436 trainer/global_step‚1699 /loss/FRt1/assignment_loss‚4.203102919033595 /loss/FRt2/assignment_loss‚4.244699771811322 1loss/FBt1/assignment_loss‚0.09769661043343755 0loss/FBt2/assignment_loss‚0.3967475798493251 /loss/FRt1/detection_loss‚4.4824814592088975 .loss/FRt2/detection_loss‚3.992944685242525 /loss/FBt2/detection_loss‚13.348643473630073 &loss/total_loss‚49.677643330484464 1loss/SRqqt1/assignment_loss‚1.386177612569674 1loss/SRqqt2/assignment_loss‚2.394697077862628 epoch‚0 _step‚135 1loss/SRqqt1/detection_loss‚1.4756580410076028 .loss/FBt1/detection_loss‚9.689233336721442‡ÿŠØÈ‚(:„ òíÄÓåéîÛ gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚120.15' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚66.75 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 æ¯~v‚(:ë òíÄÓ乜ß disk.nvme0c0n1.in‚161.5 disk.rbd1.out‚ 0.306640625 disk.sda3.out‚ 33.73828125 disk.dm-1.in‚0 disk.drbd1743.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚161.5 disk.sda.out‚ 33.73828125 disk.rbd4.in‚0 disk.dm-1.out‚ 33.73828125 disk.sda3.in‚0 proc.cpu.threads‚29 disk.dm-0.in‚161.5 disk.sda.in‚0 disk.rbd3.in‚0% disk.nvme0c0n1.out‚2983.755859375 disk.rbd1.in‚0 disk.rbd5.in‚0 disk.rbd2.out‚ 0.306640625 disk.sda1.out‚0$ memory_percent‚41.20866060256958) proc.memory.availableMB‚ 38529.4921875) disk./.usagePercent‚82.41181825603245# disk./.usageGB‚1227.99564743042 disk.rbd0.out‚0! proc.memory.rssMB‚ 11266.21875 network.sent‚ 1.540412e+06 disk.rbd5.out‚ 0.306640625 disk.sda2.in‚0 disk.rbd3.out‚0# disk.nvme0n1.out‚2983.755859375* proc.memory.percent‚17.190885543823242 network.recv‚144473 disk.rbd4.out‚ 0.36962890625 disk.sda2.out‚0 disk.rbd0.in‚0 cpu‚14.280639768379405 disk.dm-0.out‚2983.755859375 disk.drbd1743.out‚ 0.75781252S0‘+ _wandbruntime‚731  _runtime‚731?û‚qÆ’À & lr-AdamW/pg2‚4.507555416280036e-05 trainer/global_step‚1749 % _timestamp‚1.7858045420755498e+09 _runtime‚ 731.348831287 _step‚136 & lr-AdamW/pg1‚4.507555416280036e-05›fÇ{Ë“Å & lr-AdamW/pg1‚4.507555416280036e-05 & lr-AdamW/pg2‚4.507555416280036e-05 trainer/global_step‚1749 % _timestamp‚1.7858045420755498e+09 _runtime‚ 731.348831287 _step‚136ˆ-¿0”+ _wandbruntime‚731  _runtime‚731…ûu,¹'•³O WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30118303638528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 % _timestamp‚1.7858045420764852e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 trainer/global_step‚1749 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18486375 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6892145 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚30046360904704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1581457017856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31585926465536 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1539461091840 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1581317955072 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31585821996544 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11629829 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚ 731.353564199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚30046377944064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚30046360904704 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30116204 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1535778902324 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29299846237472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18486375 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1539548521472 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30835642179156 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18476719 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚31585926465536 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1539461091840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18476719 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31699760656384 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6892147 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚31585821996544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30835539336672 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11629829 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30046377944064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11629827 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30118278734848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11629827 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31699596689920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29299863276832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18<ØG¾'–¸O G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30118303638528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 % _timestamp‚1.7858045420764852e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 trainer/global_step‚1749 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18486375 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6892145 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚30046360904704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1581457017856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31585926465536 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1539461091840 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1581317955072 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31585821996544 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11629829 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚ 731.353564199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚30046377944064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚30046360904704 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30116204 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1535778902324 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29299846237472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18486375 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1539548521472 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30835642179156 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18476719 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚31585926465536 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1539461091840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18476719 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31699760656384 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6892147 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚31585821996544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30835539336672 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11629829 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30046377944064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11629827 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30118278734848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11629827 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31699596689920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29299863276832‰‚4ª0—+ _wandbruntime‚731  _runtime‚731³ Ǹù ˜½M G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30123712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18487242 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30133384 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18487242 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11915973 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5019651 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141152768 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31717706215424 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6896082 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30063544707072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30133384 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11636472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18496912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚138 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11636472 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H@DeviceStatsMonito*Êr.on_train_batch_end/segment.small_pool.current‚121 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1540340711424 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31717872271872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1582220363776 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1540428148224 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11636470 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11915733 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30853259824810 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚31603972855296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30135485851648 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚31603868379136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 trainer/global_step‚1749 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚30063527667712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30123712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 731.684789756 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024ç3V@È&™ÂM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11636470 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11915733 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30853259824810 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚31603972855296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30135485851648 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚31603868379136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 trainer/global_step‚1749 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚30063527667712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30123712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 731.684789756 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30123712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18487242 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30133384 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18487242 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11915973 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5019651 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141152768 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31717706215424 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6896082 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30063544707072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30133384 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11636472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18496912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚138 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11636472 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1540340711424 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31717872271872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1582220363776 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1540428148224 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11636470 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19004721 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚30904699055104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11961977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11961979 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30976356 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19014377 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30136854853472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11961979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1626514703872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1579667365832 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 $ _timestamp‚1.785804561925781e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30978634574848 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5159425 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30904716094464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7088995 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚32488260751360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19004721 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5159177 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚32488260751360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1583457227264 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1579581562708 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1583457227264 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚32488156282368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1626641183744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚32488156282368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30978659478528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7088997 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚30904716094464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0²úÔ‘!¦·O L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19004721 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚30904699055104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11961977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11961979 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30976356 G>DeviceStatsMonitor.on_train_bat&Ÿœ$ch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19014377 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30136854853472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11961979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1626514703872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1579667365832 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 $ _timestamp‚1.785804561925781e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30978634574848 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5159425 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30904716094464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7088995 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚32488260751360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19004721 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5159177 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚32488260751360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1583457227264 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1579581562708 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1583457227264 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚32488156282368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1626641183744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚32488156282368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30978659478528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7088997 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚30904716094464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 751.202813986 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291îÎ#0§+ _wandbruntime‚751  _runtime‚751kôÙQÃ&¨½M K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12254977 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7092934 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30153595025792 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1584336846848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1584424283648 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11968620 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5162045 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚142 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚32506307141120 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12255232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30993536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11968622 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1580544839166 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11968622 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1627414810624 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11968620 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0õ²ýbÈ&©ÂM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1580544839166 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11968622 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1627414810624 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11968620 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12254977 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7092934 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30153595025792 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1584336846848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1584424283648 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11968620 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5162045 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚142 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚32506307141120 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12255232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30993536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11968622 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0Žô¹åñ0ª+ _wandbruntime‚751  _runtime‚751‹k«û 2loss/SRqqt1/assignment_loss‚1.3546264004143118 1loss/SRqqt2/assignment_loss‚2.235780955273278 1loss/FBt1/assignment_loss‚0.13553780991706946 1loss/SRqqt1/detection_loss‚2.0882678682823315 1loss/SRqqt2/detection_loss‚2.2750975832709117 %loss/total_loss‚42.38931529558356 trainer/global_step‚1799 /loss/FRt1/assignment_loss‚4.320190758239932 0loss/FBt2/assignment_loss‚0.2874762175871927 /loss/FRt1/detection_loss‚3.7791901379823685 .loss/FRt2/detection_loss‚3.224793549703092 _runtime‚ 751.538935964 _step‚143 0loss/FRt2/assignment_loss‚3.8938794938885435 epoch‚0 % _timestamp‚1.7858045622655027e+09 .loss/FBt1/detection_loss‚6.899566300808349 /loss/FBt2/detection_loss‚11.894908220216182vÅ[Á¬€ 1loss/SRqqt2/detection_loss‚2.2750975832709117 %loss/total_loss‚42.38931529558356 trainer/global_step‚1799 /loss/FRt1/assignment_loss‚4.320190758239932 0loss/FBt2/assignment_loss‚0.2874762175871927 /loss/FRt1/detection_loss‚3.7791901379823685 .loss/FRt2/detection_loss‚3.224793549703092 _runtime‚ 751.538935964 _step‚143 0loss/FRt2/assignment_loss‚3.8938794938885435 epoch‚0 % _timestamp‚1.7858045622655027e+09 .loss/FBt1/detection_loss‚6.899566300808349 /loss/FBt2/detection_loss‚11.894908220216182 2loss/SRqqt1/assignment_loss‚1.3546264004143118 1loss/SRqqt2/assignment_loss‚2.235780955273278 1loss/FBt1/assignment_loss‚0.13553780991706946 1loss/SRqqt1/detection_loss‚2.0882678682823315lt O­‚(:„ ŸîÄÓÅáªÛ gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚152.91' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚84.95 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aìócv®‚(:ë ŸîÄÓ…°à* proc.memory.availableMB‚38529.44140625# proc.memory.rssMB‚ 11266.2265625 disk.drbd1743.in‚0 disk.rbd2.out‚ 0.306640625 disk.rbd1.out‚ 0.306640625 disk.rbd5.out‚ 0.306640625$ memory_percent‚41.20873808860779 disk.drbd1743.out‚ 0.7890625 disk.rbd2.in‚0 disk.sda1.out‚0 disk.dm-0.out‚ 2985.54296875* proc.memory.percent‚17.190897464752197 network.sent‚ 1.632681e+06 network.recv‚150096 disk.rbd3.out‚0$ disk.nvme0c0n1.out‚ 2985.54296875" disk.nvme0n1.out‚ 2985.54296875 disk.rbd0.in‚0) disk./.usagePercent‚82.44551447970422$ disk./.usageGB‚1228.497745513916 disk.sda3.out‚ 35.89453125 disk.dm-0.in‚161.5 disk.sda.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 35.89453125 disk.rbd4.in‚0 disk.dm-1.out‚ 35.89453125 disk.sda1.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚161.5 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd4.out‚ 0.36962890625 disk.rbd5.in‚0 cpu‚14.26894606801038 proc.cpu.threads‚29 disk.nvme0c0n1.in‚161.5 disk.rbd0.out‚0 disk.dm-1.in‚0„=¾0¯+ _wandbruntime‚770  _runtime‚770 «ü0Æ°À & lr-AdamW/pg1‚4.765277281133098e-05 & lr-AdamW/pg2‚4.765277281133098e-05 trainer/global_step‚1849 % _timestamp‚1.7858045816526992e+09 _runtime‚ 770.925990098 _step‚144,å#±Å trainer/global_step‚184Mñåg¨9 % _timestamp‚1.7858045816526992e+09 _runtime‚ 770.925990098 _step‚144 & lr-AdamW/pg1‚4.765277281133098e-05 & lr-AdamW/pg2‚4.765277281133098e-05‹ã÷u0²+ _wandbruntime‚770  _runtime‚770Œg3¹'³³O BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33390490568192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12294127 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚30973880508832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12588323 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚32597436338172 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚31763037205504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31763037205504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7285847 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1671836666880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12294129 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31826850 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1623555829340 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 trainer/global_step‚1849 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31763054244864 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚33390490568192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚31763054244864 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚32597333495688 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7285845 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30973863469472 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1623470026216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19532723 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12588586 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31838990414848§™4¾'´¸O N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7285845 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30973863469472 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1623470026216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19532723 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12588586 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31838990414848 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33390490568192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12294127 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚30973880508832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12588323 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚32597436338172 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚31763037205504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31763037205504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7285847 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1671836666880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12294129 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31826850 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1623555829340 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 trainer/global_step‚1849 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31763054244864 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚33390490568192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚31763054244864 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚32597333495688‘?e 0µ+ _wandbruntime‚771  _runtime‚771 Æ0Ã&¶½M A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7289782 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5305629 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30990620681152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚31780203968512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12300770 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12595156 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31853688 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1628420419072 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚33528957948416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12595413 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1628420419072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1624347499438 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1628332982272 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31844016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31853688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30990603641792 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31780203968512 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1628332982272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19552916 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5305374 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33408536950784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19543246 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚33408641426944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1672735513088 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12300770 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚32615053983826 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33408641426944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12300772 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31844016 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31780221007872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚1849 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19543246 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _runtime‚ 771.292172668 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31780221007872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1672606943232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 % _timestamp‚1.7858045820182812e+09 _step‚146 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888ž·ÂM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚31780203968512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12300770 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12595156 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31853688 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1628420419072 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚33528957948416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12595413 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1628420419072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1624347499438 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1628332982272 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 ?5DeviceStatsMonitor.on_train_batch_end/active.alûã;l.peak‚11887 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31844016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31853688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30990603641792 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31780203968512 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1628332982272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19552916 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5305374 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33408536950784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19543246 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚33408641426944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1672735513088 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12300770 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚32615053983826 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33408641426944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12300772 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31844016 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31780221007872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚1849 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19543246 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _runtime‚ 771.292172668 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31780221007872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1672606943232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 % _timestamp‚1.7858045820182812e+09 _step‚146 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7289782 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5305629 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30990620681152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640’%Êú0¸+ _wandbruntime‚771  _runtime‚771pã1Û¹ÿ 1loss/SRqqt2/detection_loss‚3.3890066404570045 .loss/FBt2/detection_loss‚8.250430211899996 % _timestamp‚1.7858045820196352e+09 1loss/FBt2/assignment_loss‚0.41705947030335666 /loss/FRt2/detection_loss‚3.4192997540837977 trainer/global_step‚1849 1loss/SRqqt2/assignment_loss‚2.143181554145283 1loss/FBt1/assignment_loss‚0.12532692534938072 1loss/SRqqt1/detection_loss‚1.8841761684304708 /loss/FBt1/detection_loss‚10.339057875646766 epoch‚0 _step‚147 0loss/FRt1/assignment_loss‚4.2667702552748885 0loss/FRt2/assignment_loss‚4.5814191833619144 2loss/SRqqt1/assignment_loss‚1.5126802887672843 /loss/FRt1/detection_loss‚3.7587204255708833 &loss/total_loss‚44.087128753291026 _runtime‚ 771.292941339÷Ï““ º„ _step‚147 0loss/FRt1/assignment_loss‚4.2667702552748885 0loss/FRt2/assignment_loss‚4.5814191833619144 2loss/SRqqt1/assignment_loss‚1.5126802887672843 /loss/FRt1/detection_loss‚3.7587204255708833 &loss/total_loss‚44.087128753291026 _runtime‚ 771.292941339 1loss/SRqqt2/detection_loss‚3.3890066404570045 .loss/FBt2/detection_loss‚8.250430211899996 % _timestamp‚1.7858045820196352e+09 1loss/FBt2/assignment_loss‚0.41705947030335666 /loss/FRt2/detection_loss‚3.4192997540837977 trainer/global_step‚1849 1loss/SRqqt2/assignment_loss‚2.143181554145283 1loss/FBt1/assignment_loss‚0.12532692534938072 1loss/SRqqt1/detection_loss‚1.8841761684304708 /loss/FBt1/detection_loss‚10.339057875646766 epoch‚0“ØT0œ»‚(:‘ ®îÄÓë¶Û gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚125.011' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚69.45055555555555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i÷£i¼‚(:Þ ®îÄÓûŸùÞ disk.dm-0.out‚ 3583.43359375 disk.sda.in‚0 disk.rbd4.out‚ 0.36962890625 disk.rbd2.out‚ 0.306640625" disk.nvme0n1.out‚ 3583.43359375 disk.sda3.out‚36.375 disk.sda2.out‚0 disk.rbd3.out‚0 network.recv‚152256% disk./.usageGB‚1228.6212921142578 disk.rbd5.out‚ 0.306640625 disk.dm-1.in‚0 disk.rbd2.in‚0* proc.memory.availableMB‚38529.87109375 disk.drbd1743.out‚ 0.81640625 disk.rbd0.out‚0 disk.sda1.out‚0$ disk.nvme0c0n1.out‚ 3583.43359375 disk.rbd1.out‚ 0.306640625 proc.cpu.threads‚29 disk.dm-0.in‚161.5 disk.sda3.in‚0 disk.nvme0c0n1.in‚161.5 disk.rbd4.in‚0 disk.drbd1743.in‚0 disk.sda1.in‚0$ memory_percent‚41.20808243751526 network.sent‚ 1.67747e+06 disk.sda.out‚36.375 disk.rbd5.in‚0 disk.dm-1.out‚36.375 disk.sda2.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 11266.2265625* proc.memory.percent‚17.190897464752197) disk./.usagePercent‚82.45380579571565 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚161.5 cpu‚14.264996285801619²ä¸Ê0½+ _wandbruntime‚790  _runtime‚790—>{DǾÁ _runtime‚ 790.413047453 _step‚148 ' lr-AdamW/pg1‚4.8941382135596276e-05 ' lr-AdamW/pg2‚4.8941382135596276e-05 trainer/global_step‚1899 $ _timestamp‚1.785804601139782e+09x1Ì¿Æ _step‚148 ' lr-AdamW/pg1‚4.8941382135596276e-05 ' lr-AdamW/pg2‚4.8941382135596276e-05 trainer/global_step‚1899 $ _timestamp‚1.785804601139782e+09 _runtime‚ 790.413047453”Ã̆Ç0À+ _wandbruntime‚790  _runtime‚790ôÌç¹'Á³O ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34292824854016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12928370 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20060725 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚32621375355904 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12626279 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚33478333417680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1667444292848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1671536927744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚149 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12626277 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12928135 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20070381 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚32699371158528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚34292929323008 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32696660 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20070381 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚33478230575196 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 _runtime‚ 790.417557235 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1671449498112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34416211240960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚34292929323008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7482697 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34416371013120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚34292824854016 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚32621392395264 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚32621392395264 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚32699346254848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1716999854592 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12626279 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5445673 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚31810889124832 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32687002 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1716864986112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20060725 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1667358489724 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 trainer/global_step‚1899 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31810872085472aåJ†¾'¸O `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚32699346254848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1716999854592 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12626279 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5445673 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚31810889124832 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32687002 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1716864986112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20060725 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1667358489724 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 trainer/global_step‚1899 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31810872085472 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34292824854016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12928370 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20060725 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚32621375355904 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12626279 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚33478333417680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1667444292848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1671536927744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚149 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12626277 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12928135 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20070381 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚32699371158528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚34292929323008 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32696660 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20070381 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚33478230575196 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 _runtime‚ 790.417557235 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1671449498112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34416211240960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚34292929323008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7482697 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34416371013120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚34292824854016 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚32621392395264 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚32621392395264•lÑ |0Ã+ _wandbruntime‚790  _runtime‚790ã»,q…ĽM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12632920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20080918 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚32638559158272 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.la‰iš>#rge_pool.freed‚32638542118912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20071248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32713840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12934870 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34434476132352 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚31827612257792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚32638542118912 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5448238 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1717897857024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20071248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5448488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1672416554496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32704168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚32716553371648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1672416554496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚31827629297152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1668321766182 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1717767190016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1668235962946 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚32638559158272 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚32716578275328 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚33495848220738 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚34310975712768 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7486632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚ 790.747107633⛥È&ÅÂM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1672416554496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚31827629297152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1668321766182 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1717767190016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1668235962946 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚32638559158272 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚32716578275328 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚33495848220738 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚34310975712768 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7486632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚ 790.747107633 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12632920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20080918 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚32638559158272 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚32638542118912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20071248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32713840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12934870 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34434476132352 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚31827612257792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚32638542118912 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5448238 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1717897857024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20071248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5448488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1672416554496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32704168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚32716553371648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121–Üñák0Æ+ _wandbruntime‚790  _runtime‚790lʇZüÇö 0loss/FBt2/assignment_loss‚0.5848198113023196 .loss/FRt2/detection_loss‚3.264795981506084 1loss/SRqqt1/detection_loss‚1.8813898325632588 %loss/total_loss‚37.54878179703213 _step‚151 /loss/FRt1/assignment_loss‚4.157141639945213 1loss/SRqqt2/assignment_loss‚2.147536445689458 1loss/SRqqt2/detection_loss‚2.4207061561270136 -loss/FBt2/detection_loss‚8.30290903527604 epoch‚0 2loss/SRqqt1/assignment_loss‚1.4758957867221585 .loss/FRt1/detection_loss‚3.443426653240504 .loss/FBt1/detection_loss‚5.886040377426306 trainer/global_step‚1899 1loss/FBt1/assignment_loss‚0.13067775326500716 % _timestamp‚1.7858046014747715e+09 _runtime‚ 790.7478639 0loss/FRt2/assignment_loss‚3.8534423239687654·ÌÇ»Èû _runtime‚ 790.7478639 0loss/FRt2/assignment_loss‚3.8534423239687654 0loss/FBt2/assignment_loss‚0.5848198113023196 .loss/FRt2/detection_loss‚3.264795981506084 1loss/SRqqt1/detection_loss‚1.8813898325632588 %loss/total_loss‚37.54878179703213 _step‚151 /loss/FRt1/assignment_loss‚4.157141639945213 1loss/SRqqt2/assignment_loss‚2.147536445689458 1loss/SRqqt2/detection_loss‚2.4207061561270136 -loss/FBt2/detection_loss‚8.30290903527604 epoch‚0 2loss/SRqqt1/assignment_loss‚1.4758957867221585 .loss/FRt1/detection_loss‚3.443426653240504 .loss/FBt1/detection_loss‚5.886040377426306 trainer/global_step‚1899 1loss/FBt1/assignment_loss‚0.13067775326500716 % _timestamp‚1.7858046014747715e+09—¤÷Â;œÉ‚(:‘ ½îÄÓÙ›ÑÛ gpu.0.gpu‚42.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚120.979' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚67.21055555555554 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KAê…Ê‚(:ú ½îÄÓ˜ÕéÞ# proc.memory.rssMB‚ 11266.2265625% disk./.usageGB‚1228.6207122802734 disk.sda3.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.drbd1743.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 161.50390625) disk./.usagePercent‚82.45376688257045 disk.dm-0.in‚ 161.50390625 disk.rbd5.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚ 37.28515625 disk.rbd5.out‚ 0.306640625 disk.nvme0n1.in‚ 161.50390625 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.dm-0.out‚ 3708.828125 disk.rbd4.in‚0 disk.drbd1743.out‚ 0.8359375 disk.rbd1.out‚ 0.306640625% memory_percent‚41.208577156066895 disk.sda3.out‚ 37.28515625 disk.rbd0.in‚0( proc.memory.availableMB‚ 38529.546875* proc.memory.percent‚17.190897464752197 proc.cpu.threads‚29 disk.dm-1.out‚ 37.28515625 disk.rbd2.out‚ 0.306640625 disk.sda1.out‚0" disk.nvme0c0n1.out‚ 3708.828125 cpu‚14.26169154323111 network.sent‚ 1.722194e+06 network.recv‚154697 disk.rbd4.out‚ 0.36962890625 disk.rbd3.in‚0 disk.nvme0n1.out‚ 3708.828125=H0Ë+ _wandbruntime‚809  _runtime‚809墤ÅÌ¿ _step‚152 & lr-AdamW/pg1‚5.022999145986158e-05 & lr-AdamW/pg2‚5.022999145986158e-05 trainer/global_step‚1949 % _timestamp‚1.7858046206617267e+09 _runtime‚ 809.93499689 Á©€ÊÍÄ & lr-AdamW/pg1‚5.022999145986158e-05 & lr-AdamW/pg2‚5.022999145986158e-05 trainer/global_step‚1949 % _timestamp‚1.7858046206617267e+09 _runtime‚ 809.93499689 _step‚152˜¡c_0Î+ _wandbruntime‚809  _runtime‚809¥÷V¤i'ϲO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1762170735104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1762033769472 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33559726998528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12958427 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12958429 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚35195159139840 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35195159139840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7679547 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5588667 $ _timestamp‚1.785804620662657e+09 _step‚153 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚33479713506304 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1715533063168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚1949 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1711332756356 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚35321735864320 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33556812 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1711246953232 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20598383 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20588727 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5588922 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚32647880701472 _runtime‚ 809.939556063 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚35195263608832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1715533063168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚35195263608832 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20598383 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33479730545664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13268469 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚32647897740832 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚33479713506304 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚34359127654704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12958427 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7679545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1715445633536 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ]JDeviceStatsM®ÂoÂOonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚35321897733632™M‡5½'зO trainer/global_step‚1949 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1711332756356 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚35321735864320 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33556812 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1711246953232 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20598383 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20588727 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5588922 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚32647880701472 _runtime‚ 809.939556063 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚35195263608832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1715533063168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚35195263608832 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20598383 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33479730545664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13268469 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚32647897740832 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚33479713506304 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚34359127654704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12958427 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7679545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1715445633536 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚35321897733632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1762170735104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1762033769472 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33559726998528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12958427 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12958429 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚35195159139840 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35195159139840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7679547 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5588667 $ _timestamp‚1.785804620662657e+09 _step‚153 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚33479713506304 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1715533063168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640™Ù}‡ œÑ‚(:‘ ÌîÄÓ«×Û gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚119.721' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚66.51166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç£‰G‹Ò‚(:€  ÌîÄÓº¥ûÞ network.sent‚ 1.766824e+06 disk.rbd5.in‚0 disk.rbd5.out‚ 0.306640625 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0* proc.memory.percent‚17.190897464752197 disk.dm-0.out‚ 3712.82421875 disk.dm-1.out‚ 37.81640625 disk.nvme0n1.in‚ 161.50390625" disk.nvme0n1.out‚ 3712.82421875 network.recv‚156593 disk.rbd2.out‚ 0.306640625 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 161.50390625 disk.rbd1.in‚0 disk.rbd0.in‚0% disk./.usageGB‚1228.6251945495605 disk.rbd4.in‚0 disk.drbd1743.in‚0 disk.rbd0.out‚0* proc.memory.availableMB‚38528.42578125 cpu‚14.258760196437581$ disk.nvme0c0n1.out‚ 3712.82421875 disk.rbd4.out‚ 0.36962890625 disk.rbd3.out‚0# proc.memory.rssMB‚ 11266.2265625 proc.cpu.threads‚29) disk./.usagePercent‚82.45406769142318 disk.rbd1.out‚ 0.306640625 disk.sda3.out‚ 37.81640625$ memory_percent‚41.21028780937195 disk.dm-0.in‚ 161.50390625 disk.sda.in‚0 disk.sda.out‚ 37.81640625 disk.drbd1743.out‚0.84375 disk.rbd2.in‚0ÐìcO0Ó+ _wandbruntime‚810  _runtime‚810Í\Ã&Ô½M SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12965072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1716412689920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33496897308672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1716325253120 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚35213205522432 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚34376745300246 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚32664637913152 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33564320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1716325253120 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7683484 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 % _timestamp‚1.7858046210028827e+09 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33564320 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5591761 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20608920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1763071981568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚35339844234752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚34376848142842 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12965072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1762935023104 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13275010 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33573992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 trainer/global_step‚1949 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35340006096896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12965070 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33576909211648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33496880269312 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚33496880269312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712eüS1È&ÕÂM S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚35213309998592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1716412689920 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5591528 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚32664620873792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12965070 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33576909211648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33496880269312 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚33496880269312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12965072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1716412689920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33496897308672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1716325253120 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚35213205522432 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚34376745300246 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚32664637913152 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33564320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1716325253120 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7683484 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 % _timestamp‚1.7858046210028827e+09 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33564320 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5591761 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20608920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1763071981568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚35339844234752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚34376848142842 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12965072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1762935023104 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13275010 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33573992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 trainer/global_step‚1949 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35340006096896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ODeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚36097597894656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚35240024734212 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚33484889317472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚36097597894656 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚36097493425664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1759441768960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚34338051656704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1807294658560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13290577 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13290579 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7876397 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13290579 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1759529198592 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚35240127576696 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1755221219864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚34338068696064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34420057934848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1807163984384 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36227377497088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21126385 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13608996 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1759441768960 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992NM ž'à¸O TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚36097493425664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21116729 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚36097597894656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚35240024734212 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚33484889317472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚36097597894656 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚36097493425664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1759441768960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚34338051656704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1807294658560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13290577 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13290579 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7876397 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13290579 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1759529198592 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚35240127576696 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1755221219864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚34338068696064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34420057934848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1807163984384 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36227377497088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21126385 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13608996 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1759441768960 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13290577 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7876395 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0+0á+ _wandbruntime‚830  _runtime‚830òƒÙ“%â»M OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34424472 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33501646529152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13615726 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5735392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚35257642379754 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚34355218419712 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1760408825344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1808078795776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34424472 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚36115644284416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36245503612416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 trainer/global_step‚1999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚34355235459072 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5735149 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 # _timestamp‚1.78580464074356e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1760321388544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7880334 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚36115539808256 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7880332 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚34355218419712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13297220 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13297220 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚36115644284416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1808213657088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1760321388544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13297222 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21136922 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21127252 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34434144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚34437265051648 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚35257745222350 _runtime‚ 830.017149073 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1760408825344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13297222 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1756098693198 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21127252 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13615481 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34434144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21136922 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_devic|¦÷.e_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0(¹†Æ&ãÀM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚34437265051648 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚35257745222350 _runtime‚ 830.017149073 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1760408825344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13297222 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1756098693198 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21127252 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13615481 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34434144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21136922 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34424472 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33501646529152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13615726 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5735392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚35257642379754 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚34355218419712 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1760408825344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1808078795776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34424472 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚36115644284416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36245503612416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 trainer/global_step‚1999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚34355235459072 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5735149 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 # _timestamp‚1.78580464074356e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1760321388544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7880334 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚36115539808256 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7880332 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚34355218419712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13297220 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13297220 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚36115644284416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1808213657088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1760321388544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13297222 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21136922 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21127252 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34434144ž›5ë0ä+ _wandbruntime‚830  _runtime‚830®©G¢åú /loss/FRt1/assignment_loss‚4.111622339271637 /loss/FRt2/assignment_loss‚3.998147214966259 0loss/FBt1/assignment_loss‚0.1499155898030343 /loss/FRt2/detection_loss‚3.5564100244778327 1loss/SRqqt2/detection_loss‚3.8915068393201557 .loss/FBt2/detection_loss‚9.721966871158472 _runtime‚ 830.017824208 _step‚159 1loss/SRqqt1/assignment_loss‚1.407471753046161 2loss/SRqqt2/assignment_loss‚1.8882599101519921 1loss/FBt2/assignment_loss‚0.30201099740990783 .loss/FBt1/detection_loss‚3.620651516294295 trainer/global_step‚1999 /loss/FRt1/detection_loss‚3.6209972755736617 % _timestamp‚1.7858046407447257e+09 1loss/SRqqt1/detection_loss‚1.6505492699897868 %loss/total_loss‚37.91950960146319 epoch‚0»¦µ æÿ /loss/FRt2/assignment_loss‚3.998147214966259 0loss/FBt1/assignment_loss‚0.1499155898030343 /loss/FRt2/detection_loss‚3.5564100244778327 1loss/SRqqt2/detection_loss‚3.8915068393201557 .loss/FBt2/detection_loss‚9.721966871158472 _runtime‚ 830.017824208 _step‚159 1loss/SRqqt1/assignment_loss‚1.407471753046161 2loss/SRqqt2/assignment_loss‚1.8882599101519921 1loss/FBt2/assignment_loss‚0.30201099740990783 .loss/FBt1/detection_loss‚3.620651516294295 trainer/global_step‚1999 /loss/FRt1/detection_loss‚3.6209972755736617 % _timestamp‚1.7858046407447257e+09 1loss/SRqqt1/detection_loss‚1.6505492699897868 %loss/total_loss‚37.91950960146319 epoch‚0 /loss/FRt1/assignment_loss‚4.111622339271637Ÿ?£gcœç‚(:‘ êîÄÓÈÉâÛ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚120.252' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚66.80666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª×'Ýè‚(:…  êîÄÓ•¿Óß disk.drbd1743.out‚ 0.87890625# disk.nvme0n1.out‚3713.869140625 disk.rbd4.in‚0$ memory_percent‚41.21071100234985% disk.nvme0c0n1.out‚3713.869140625 disk.sda3.out‚ 39.36328125 disk.rbd5.in‚0 disk.drbd1743.in‚0* proc.memory.percent‚17.190897464752197 disk.rbd2.out‚ 0.306640625 disk.sda1.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 39.36328125) proc.memory.availableMB‚ 38528.1484375 cpu‚14.251575668545005 proc.cpu.threads‚29 disk.rbd1.out‚ 0.306640625 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd4.out‚ 0.36962890625 network.sent‚ 1.835152e+06) disk./.usagePercent‚82.45406871545332% disk./.usageGB‚1228.6252098083496 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 161.53515625 disk.dm-1.in‚0# proc.memory.rssMB‚ 11266.2265625 disk.nvme0n1.in‚ 161.53515625 disk.dm-0.in‚ 161.53515625 disk.dm-0.out‚3713.869140625 disk.sda.in‚0 network.recv‚160753 disk.sda2.out‚0 disk.rbd5.out‚ 0.306640625 disk.rbd1.in‚0 disk.sda.out‚ 39.36328125 disk.rbd3.out‚0¬W·ò0é+ _wandbruntime‚849  _runtime‚849>ÌåÄê¾ _step‚160 & lr-AdamW/pg1‚5.280721010839219e-05 & lr-AdamW/pg2‚5.280721010839219e-05 trainer/global_step‚2049 $ _timestamp‚1.785804660164396e+09 _runtime‚ 849.43767362äö Éëà _step‚160 & lr-AdamW/pg1‚5.280721010839219e-05 & lr-AdamW/pg2‚5.280721010839219e-05 trainer/global_step‚2049 $ _timestamp‚1.785804660164396e+09 _runtime‚ 849.43767362 w[å0ì+ _wandbruntime‚849  _runtime‚849¶¤¹'í³O A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36121024656204 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1803525334016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13622727 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35280413774848 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35267458 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21654387 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚36999827711488 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚36999827711488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚35196389807104 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚35196406846464 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1803437904384 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚34321914972832 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13948570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35280438678528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1852361030144 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36120921813720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21644731 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7858046601653185e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1799023880248 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚35196406846464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚34321897933472 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37132774804992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37132936674304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5875323 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 trainer/global_step‚2049 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚35196389807104 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13622727 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13948310 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1852497995776 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1803437904384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13622729 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8073245 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1803525334016 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚36999932180480 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35277116 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1799109683372 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚36999827711488 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚36999827711488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚35196389807104 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚35196406846464 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1803437904384 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚34321914972832 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13948570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35280438678528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1852361030144 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36120921813720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21644731 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7858046601653185e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1799023880248 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚35196406846464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚34321897933472 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37132774804992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37132936674304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5875323 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 trainer/global_step‚2049 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚35196389807104 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13622727 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13948310 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1852497995776 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool¶æR.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1803437904384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13622729 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8073245 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1803525334016 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚36999932180480 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35277116 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1799109683372 IKÃ&ð½M E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7858046604983835e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5878142 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1804317523968 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13955079 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚34338655145152 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35284624 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚35213573609472 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 _step‚162 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37017874094080 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1853397300736 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37017978570240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35297620891648 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21664924 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13629370 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21655254 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37017978570240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35284624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35297645795328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚ 849.772160655 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13629372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8077184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37151043096064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1804317523968 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35294296 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚36138642301858 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1799901353470 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚34338638105792 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21655254 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13629372 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21664924 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36138539459262 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1799987156706 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35294296 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1804404960768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160è€ýÈ&ñÂM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21664924 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13629370 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21655254 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37017978570240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35284624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35297645795328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚ 849.772160655 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13629372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8077184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37151043096064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1804317523968 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35294296 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚36138642301858 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1799901353470 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚34338638105792 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21655254 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13629372 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21664924 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36138539459262 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1799987156706 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35294296 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1804404960768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7858046604983835e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5878142 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1804317523968 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13955079 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚34338655145152 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35284624 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚35213573609472 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 _step‚162 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37017874094080 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1853397300736 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37017978570240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35297620891648¢¼´3•0ò+ _wandbruntime‚849  _runtime‚849]~™þóø 0loss/SRqqt2/detection_loss‚3.400791784494072 .loss/FBt1/detection_loss‚7.996198601380456 epoch‚0 trainer/global_step‚2049 _step‚163 1loss/FBt1/assignment_loss‚0.03584021780765257 %loss/total_loss‚44.50476236045819 % _timestamp‚1.7858046605001786e+09 _runtime‚ 849.773283178 /loss/FRt2/assignment_loss‚4.292704835042849 1loss/SRqqt2/assignment_loss‚1.958682823394026 .loss/FRt1/detection_loss‚4.411481876264919 .loss/FRt2/detection_loss‚3.354637235894308 0loss/SRqqt1/detection_loss‚1.801199074375533 /loss/FRt1/assignment_loss‚3.980380679621841 2loss/SRqqt1/assignment_loss‚1.7702342428393283 1loss/FBt2/assignment_loss‚0.18745342696856823 /loss/FBt2/detection_loss‚11.315157562374633Ý ëôý /loss/FRt2/assignment_loss‚4.292704835042849 1loss/SRqqt2/assignment_loss‚1.958682823394026 .loss/FRt1/detection_loss‚4.411481876264919 .loss/FRt2/detection_loss‚3.354637235894308 0loss/SRqqt1/detection_loss‚1.801199074375533 /loss/FRt1/assignment_loss‚3.980380679621841 2loss/SRqqt1/assignment_loss‚1.7702342428393283 1loss/FBt2/assignment_loss‚0.18745342696856823 /loss/FBt2/detection_loss‚11.315157562374633 0loss/SRqqt2/detection_loss‚3.400791784494072 .loss/FBt1/detection_loss‚7.996198601380456 epoch‚0 trainer/global_step‚2049 _step‚163 1loss/FBt1/assignment_loss‚0.03584021780765257 %loss/total_loss‚44.50476236045819 % _timestamp‚1.7858046605001786e+09 _runtime‚ 849.773283178£OÍæÃœõ‚(:‘ ùîÄÓÀµ°Û gpu.0.gpu‚35.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚129.329' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚71.84944444444444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06Sîö‚(:…  ùîÄÓÏÒÿÞ disk.sda3.in‚0 disk.dm-1.out‚ 40.1953125 disk.rbd1.out‚ 0.306640625* proc.memory.percent‚17.190897464752197) disk./.usagePercent‚82.45407281157387% disk./.usageGB‚1228.6252708435059 disk.sda3.out‚ 40.1953125 disk.sda.in‚0 disk.rbd5.out‚ 0.306640625 disk.drbd1743.in‚0 proc.cpu.threads‚29 disk.rbd4.out‚ 0.36962890625 disk.rbd5.in‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 161.53515625 disk.rbd1.in‚0 cpu‚14.248573732389405" disk.dm-0.out‚3714.57177734375 disk.sda1.in‚0 disk.rbd3.out‚0 network.sent‚ 1.880015e+06 disk.rbd4.in‚0 disk.sda1.out‚0 network.recv‚163323 disk.rbd2.out‚ 0.306640625 disk.sda2.out‚0% disk.nvme0n1.out‚3714.57177734375' proc.memory.availableMB‚ 38527.96875 disk.dm-0.in‚ 161.53515625 disk.sda.out‚ 40.1953125 disk.rbd2.in‚0# proc.memory.rssMB‚ 11266.2265625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.drbd1743.out‚ 0.8984375 disk.rbd3.in‚0' disk.nvme0c0n1.out‚3714.57177734375 disk.nvme0n1.in‚ 161.53515625$ memory_percent‚41.21098518371582Ì0ÂÚ0÷+ _wandbruntime‚868  _runtime‚868K†©Âø¼ % lr-AdamW/pg1‚5.40958194326575e-05 % lr-AdamW/pg2‚5.40958194326575e-05 trainer/global_step‚2099 # _timestamp‚1.78580467953279e+09 _runtime‚ 868.806062241 _step‚164$3ÇùÁ % lr-AdamW/pg1‚5.40958194326575e-05 % lr-AdamW/pg2‚5.40958194326575e-05 trainer/global_step‚2099 # _timestamp‚1.78580467953279e+09 _runtime‚ 868.806062241 _step‚164¤ÖG00ú+ _wandbruntime‚868  _runtime‚868ê]Ö”‰û³O L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13954877 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37902266466304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1897667139584 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22182389 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36137268 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36127610 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13954879 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37902266466304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _runtime‚ 868.810557815 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1842912343756 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚37001818893228 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1842998146880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37902161997312 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13954877 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35158923588832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36054727957504 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 % _timestamp‚1.7858046795337167e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1847434039808 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1847434039808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚37001921735712 WKD8DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36054744996864 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37902161997312 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13954877 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35158923588832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36054727957504 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 % _timestamp‚1.7858046795337167e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1847434039808 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1847434039808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚37001921735712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6018764 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14288623 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 trainer/global_step‚2099 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1897532271104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36054744996864 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14295367 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13961522 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8274032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚35175663761152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36154448 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36154448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1843789816978 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13961522 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36071911759872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36071911759872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22192926 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36071894720512 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚38056583620096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38056421757952 # _timestamp‚1.78580467986906e+09 _step‚166 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22183256 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37920208379904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1848313659392 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1898445026304 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 _runtime‚ 869.142680816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚36158001635328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36071894720512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36157976731648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14295625 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1848401096192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22183256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22192926 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35175646721792 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6021591 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1898581984768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18/¸rpÆ&ÿÀM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36071894720512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36157976731648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14295625 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1848401096192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22183256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22192926 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35175646721792 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6021591 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1898581984768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1848313659392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14295367 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13961522 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8274032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚35175663761152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36154448 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36154448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1843789816978 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13961522 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36071911759872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36071911759872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22192926 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36071894720512 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚38056583620096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38056421757952 # _timestamp‚1.78580467986906e+09 _step‚166 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22183256 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37920208379904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1848313659392 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1898445026304 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 _runtime‚ 869.142680816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚36158001635328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360¦€r'0€+ _wandbruntime‚869  _runtime‚869fçígú /loss/FRt1/detection_loss‚3.3942159034054855 %loss/total_loss‚37.39053083234308 epoch‚0 $ _timestamp‚1.785804679870241e+09 2loss/SRqqt2/assignment_loss‚1.8051308265990682 0loss/FBt2/assignment_loss‚0.4479231719839542 .loss/FBt1/detection_loss‚7.394079123337537 trainer/global_step‚2099 _runtime‚ 869.143676552 _step‚167 0loss/FRt1/assignment_loss‚3.9924186646253212 2loss/SRqqt1/assignment_loss‚1.6387266705207753 1loss/FBt1/assignment_loss‚0.02234905372426424 /loss/FRt2/detection_loss‚3.5904694874657364 0loss/SRqqt1/detection_loss‚1.479369697765606 0loss/SRqqt2/detection_loss‚2.766558930180084 /loss/FBt2/detection_loss‚6.7010876227844465 /loss/FRt2/assignment_loss‚4.158201679950807.¼²‚ÿ _runtime‚ 869.143676552 _step‚167 0loss/FRt1/assignment_loss‚3.9924186646253212 2loss/SRqqt1/assignment_loss‚1.6387266705207753 1loss/FBt1/assignment_loss‚0.02234905372426424 /loss/FRt2/detection_loss‚3.5904694874657364 0loss/SRqqt1/detection_loss‚1.479369697765606 0loss/SRqqt2/detection_loss‚2.766558930180084 /loss/FBt2/detection_loss‚6.7010876227844465 /loss/FRt2/assignment_loss‚4.158201679950807 /loss/FRt1/detection_loss‚3.3942159034054855 %loss/total_loss‚37.39053083234308 epoch‚0 $ _timestamp‚1.785804679870241e+09 2loss/SRqqt2/assignment_loss‚1.8051308265990682 0loss/FBt2/assignment_loss‚0.4479231719839542 .loss/FBt1/detection_loss‚7.394079123337537 trainer/global_step‚2099§¿1{›ƒ‚(: ˆïÄÓÍœ°Û gpu.0.gpu‚44.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚151.97' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚84.42777777777778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W3»“„‚(:ˆ  ˆïÄÓž¡íÞ disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda3.out‚ 40.74609375) proc.memory.availableMB‚ 38527.4453125* proc.memory.percent‚17.190897464752197 proc.cpu.threads‚29" disk.dm-0.out‚3714.75927734375' disk.nvme0c0n1.out‚3714.75927734375 cpu‚14.245776446700308 disk.sda.in‚0 disk.sda.out‚ 40.74609375 disk.rbd5.out‚ 0.306640625 disk.dm-1.out‚ 40.74609375 disk.drbd1743.out‚0.90625 disk.rbd2.out‚ 0.306640625 disk.rbd4.in‚0 disk.rbd4.out‚ 0.36962890625 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 161.53515625 disk.sda3.in‚0 network.recv‚165410 disk.sda2.out‚0 disk.nvme0n1.in‚ 161.53515625 disk.rbd0.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚3714.75927734375# proc.memory.rssMB‚ 11266.2265625) disk./.usagePercent‚82.45407460362661 disk.rbd5.in‚0 disk.dm-1.in‚0$ memory_percent‚41.21178388595581 network.sent‚ 1.924923e+06% disk./.usageGB‚1228.6252975463867 disk.rbd0.out‚0 disk.dm-0.in‚ 161.53515625 disk.drbd1743.in‚0ãgˆ½œ…‚(:‘ —ïÄÓŸ€ŠÞ gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚149.304' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚82.94666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ  “†‚(:ˆ  —ïÄÓÿȈß network.sent‚ 1.969583e+06" disk.dm-0.out‚3715.15869140625 disk.rbd2.in‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 161.53515625 disk.rbd1.in‚0 disk.nvme0n1.in‚ 161.53515625 disk.rbd0.in‚0 disk.rbd5.in‚0 disk.rbd1.out‚ 0.306640625 disk.rbd0.out‚0$ memory_percent‚41.21175408363342* proc.memory.availableMB‚38527.46484375# proc.memory.rssMB‚ 11266.2265625 cpu‚14.242421849833123 disk.sda.out‚ 41.3515625 disk.rbd4.out‚ 0.36962890625 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd4.in‚0 disk.drbd1743.in‚0 disk.sda3.in‚0' disk.nvme0c0n1.out‚3715.15869140625) disk./.usagePercent‚82.45407485963415$ disk./.usageGB‚1228.625301361084 disk.dm-0.in‚ 161.53515625 disk.sda.in‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚3715.15869140625 proc.cpu.threads‚29 disk.dm-1.out‚ 41.3515625 disk.drbd1743.out‚ 0.91015625 disk.sda1.in‚0 disk.sda2.in‚0 network.recv‚167722 disk.rbd5.out‚ 0.306640625 disk.dm-1.in‚0 disk.rbd2.out‚ 0.306640625 disk.sda3.out‚ 41.3515625* proc.memory.percent‚17.190897464752197Ý’ˆÅ0‡+ _wandbruntime‚888  _runtime‚888¡û˜fȈ ' lr-AdamW/pg1‚5.5384428756922805e-05 ' lr-AdamW/pg2‚5.5384428756922805e-05 trainer/global_step‚2149 % _timestamp‚1.7858046991574793e+09 _runtime‚ 888.430795059 _step‚168nN3Í‰Ç ' lr-AdamW/pg1‚5.5384428756922805e-05 ' lr-AdamW/pg2‚5.5384428756922805e-05 trainer/global_step‚2149 % _timestamp‚1.7858046991574793e+09 _runtime‚ 888.430795059 _step‚168¨ÇåN/0Š+ _wandbruntime‚888  _runtime‚888ï&\Ÿ¹'‹³O PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22710391 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚38804496283136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1886886610388 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14287029 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _runtime‚ 888.435573033 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 120188416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36913083147264 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1891517604864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1891517604864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14629608 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38943853768192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22700735 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35995932204832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚38804600752128 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1942848501760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚37882818815220 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14287029 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1942728313344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚37882715972736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1886800807264 trainer/global_step‚2149 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8466945 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1891430175232 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚38804600752128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38804496283136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36913066107904 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37001125454848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1891430175232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36913066107904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚38943998860288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35995915165472 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8466947 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658GÂ÷¾'Œ¸O QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37001125454848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1891430175232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36913066107904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚38943998860288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35995915165472 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8466947 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22710391 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚38804496283136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1886886610388 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14287029 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _runtime‚ 888.435573033 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 120188416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36913083147264 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1891517604864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1891517604864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14629608 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38943853768192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22700735 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35995932204832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚38804600752128 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1942848501760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚37882818815220 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14287029 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1942728313344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚37882715972736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1886800807264 trainer/global_step‚2149 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8466945 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1891430175232 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚38804600752128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38804496283136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36913066107904 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251©½ÓPê0+ _wandbruntime‚888  _runtime‚888/ÖÕ Ž½M ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14293672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1892397231616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37018357475328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚38822647141888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37014600 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8470884 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37014600 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14293670 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8470882 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6165252 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37004928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚38822647141888 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚38962119405056 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1892397231616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1892309794816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14293672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37018332571648 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22720928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36930232870912 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37004928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1887764083722 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22711258 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1943761929728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _runtime‚ 888.767917847 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚36012672377152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22711258 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1887678280486 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36930249910272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚2149 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14636134 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1943631262720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37900333618278 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36930249910272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36012655337792W¿ÁÈ&ÂM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1892397231616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1892309794816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14293672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37018332571648 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22720928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36930232870912 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37004928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1887764083722 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22711258 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1943761929728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _runtime‚ 888.767917847 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚36012672377152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22711258 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1887678280486 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36930249910272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚2149 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14636134 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1943631262720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37900333618278 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36930249910272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36012655337792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14293672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1892397231616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37018357475328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚38822647141888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37014600 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8470884 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37014600 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14293670 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8470882 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6165252 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37004928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚38822647141888 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚38962119405056 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032ª² ³!0+ _wandbruntime‚888  _runtime‚888Öµ…Gü‘ö .loss/FBt1/detection_loss‚4.732732410682357 % _timestamp‚1.7858046994954686e+09 _runtime‚ 888.768558207 2loss/SRqqt2/assignment_loss‚1.8262479920750079 /loss/FRt2/detection_loss‚3.3031608680771156 1loss/SRqqt1/detection_loss‚1.6704110446721978 epoch‚0 2loss/SRqqt1/assignment_loss‚1.3989409547930043 0loss/FBt1/assignment_loss‚0.2133922643476962 %loss/total_loss‚35.58632507465879 trainer/global_step‚2149 _step‚171 .loss/FRt1/assignment_loss‚4.05550518973929 .loss/FBt2/detection_loss‚5.588731172320154 .loss/FRt2/assignment_loss‚4.26774938269095 1loss/FBt2/assignment_loss‚0.27070098394324305 .loss/FRt1/detection_loss‚4.704718305439245 /loss/SRqqt2/detection_loss‚3.55403450587853ô†õ´’û .loss/FRt1/assignment_loss‚4.05550518973929 .loss/FBt2/detection_loss‚5.588731172320154 .loss/FRt2/assignment_loss‚4.26774938269095 1loss/FBt2/assignment_loss‚0.27070098394324305 .loss/FRt1/detection_loss‚4.704718305439245 /loss/SRqqt2/detection_loss‚3.55403450587853 .loss/FBt1/detection_loss‚4.732732410682357 % _timestamp‚1.7858046994954686e+09 _runtime‚ 888.768558207 2loss/SRqqt2/assignment_loss‚1.8262479920750079 /loss/FRt2/detection_loss‚3.3031608680771156 1loss/SRqqt1/detection_loss‚1.6704110446721978 epoch‚0 2loss/SRqqt1/assignment_loss‚1.3989409547930043 0loss/FBt1/assignment_loss‚0.2133922643476962 %loss/total_loss‚35.58632507465879 trainer/global_step‚2149 _step‚171«"ÖÈ0œ“‚(:‘ ¦ïÄÓפàÛ gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚149.545' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚83.08055555555555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô–O’”‚(:‡  ¦ïÄÓÞÌŸß disk.dm-1.in‚0 disk.drbd1743.out‚0.9375 disk.rbd2.in‚0* proc.memory.percent‚17.190897464752197 disk.rbd1.out‚ 0.306640625 disk.sda3.in‚0 disk.rbd4.in‚0 disk.rbd5.in‚0 disk.dm-1.out‚ 41.94921875# proc.memory.rssMB‚ 11266.2265625 disk.sda.out‚ 41.94921875 disk.drbd1743.in‚0 disk.sda2.out‚0 cpu‚14.24004060564853 proc.cpu.threads‚29 network.sent‚ 1.971591e+06" disk.nvme0c0n1.in‚ 161.53515625' disk.nvme0c0n1.out‚3715.44775390625% disk.nvme0n1.out‚3715.44775390625" disk.dm-0.out‚3715.44775390625$ memory_percent‚41.21137261390686 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 161.53515625* proc.memory.availableMB‚38527.71484375 disk.rbd1.in‚0 disk.dm-0.in‚ 161.53515625 disk.rbd4.out‚ 0.36962890625) disk./.usagePercent‚82.37858310192982 disk.sda3.out‚ 41.94921875 disk.rbd5.out‚ 0.306640625 disk.rbd2.out‚ 0.306640625 disk.rbd3.in‚0 network.recv‚168740% disk./.usageGB‚1227.5004196166992 disk.sda.in‚0Ne.0•+ _wandbruntime‚907  _runtime‚907ÃxDÄ–¾ % lr-AdamW/pg1‚5.66730380811881e-05 % lr-AdamW/pg2‚5.66730380811881e-05 trainer/global_step‚2199 % _timestamp‚1.7858047183353257e+09 _runtime‚ 907.610582468 _step‚172™žRÉ—Ã _step‚172 % lr-AdamW/pg1‚5.66730380811881e-05 % lr-AdamW/pg2‚5.66730380811881e-05 trainer/global_step‚2199 % _timestamp‚1.7858047183353257e+09 _runtime‚ 907.610582468¬Td£Ä0˜+ _wandbruntime‚907  _runtime‚907E×í¸'™²O YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8663795 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1987915663360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39849556730368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚36832940820832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚37771404258304 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1935513740288 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1935426310656 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37771421297664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23238393 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚39706935037952 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39706830568960 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37771421297664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23228737 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23228737 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚39706935037952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 _step‚173 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14619179 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚37771404258304 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23238393 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1988050531840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚39849396958208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1930775073896 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37861506198528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1935513740288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1935426310656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14619177 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14619177 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚39706830568960 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37861481294848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.785804718335985e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚38763613052244 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14969692 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1930775073896 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37861506198528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1935513740288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1935426310656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14619177 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14619177 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚39706830568960 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37861481294848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.785804718335985e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚38763613052244 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14969692 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚37771404258304 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1935513740288 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_poolÔ ç¯.freed‚ 1935426310656 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37771421297664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23238393 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚39706935037952 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39706830568960 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37771421297664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23228737 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23228737 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚39706935037952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 _step‚173 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14619179 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚37771404258304 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23238393 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1988050531840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚39849396958208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39867668739072 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚39724876951552 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14976450 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39867500585472 _step‚174 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37865080 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1936305930240 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168153600 $ _timestamp‚1.785804718670641e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37878713315328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14625820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8667734 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚38781230697786 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14625822 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1936393367040 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23239260 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 143249920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1931652547230 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚37788588060672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1936305930240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37878688411648 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37874752 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚37788571021312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6308718 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23248930 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚38781333540382 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36849663953792 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _runtime‚ 907.944259076 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39724981427712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37865080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚37788588060672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚37788571021312 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39724981427712 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37874752 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚38781230697786 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14625822 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1936393367040 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23239260 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 143249920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1931652547230 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚37788588060672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1936305930240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37878688411648 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37874752 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚37788571021312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6308718 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23248930 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚38781333540382 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36849663953792 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _runtime‚ 907.944259076 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39724981427712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37865080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚37788588060672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚37788571021312 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39724981427712 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37874752 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39867668739072 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚39724876951552 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14976450 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39867500585472 _step‚174 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37865080 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1936305930240 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168153600 $ _timestamp‚1.785804718670641e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37878713315328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14625820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121®KYÄh0ž+ _wandbruntime‚907  _runtime‚907Ùq(µŸü %loss/total_loss‚35.34882274531212 trainer/global_step‚2199 /loss/FRt1/detection_loss‚3.5713377120318235 _runtime‚ 907.945393212 /loss/FRt2/detection_loss‚3.1397417366504667 epoch‚0 1loss/SRqqt2/assignment_loss‚2.032592983214886 1loss/FBt2/assignment_loss‚0.23418681495271207 1loss/SRqqt2/detection_loss‚3.7062403162980027 .loss/FBt2/detection_loss‚5.747796405943536 % _timestamp‚1.7858047186722639e+09 _step‚175 /loss/FRt1/assignment_loss‚4.092364473107421 0loss/FRt2/assignment_loss‚4.2815534729706615 2loss/SRqqt1/assignment_loss‚1.4256716141066814 1loss/FBt1/assignment_loss‚0.14495932628653527 1loss/SRqqt1/detection_loss‚2.0015585171762726 .loss/FBt1/detection_loss‚4.970819372573123 Å   /loss/FRt2/detection_loss‚3.1397417366504667 epoch‚0 1loss/SRqqt2/assignment_loss‚2.032592983214886 1loss/FBt2/assignment_loss‚0.23418681495271207 1loss/SRqqt2/detection_loss‚3.7062403162980027 .loss/FBt2/detection_loss‚5.747796405943536 % _timestamp‚1.7858047186722639e+09 _step‚175 /loss/FRt1/assignment_loss‚4.092364473107421 0loss/FRt2/assignment_loss‚4.2815534729706615 2loss/SRqqt1/assignment_loss‚1.4256716141066814 1loss/FBt1/assignment_loss‚0.14495932628653527 1loss/SRqqt1/detection_loss‚2.0015585171762726 .loss/FBt1/detection_loss‚4.970819372573123 %loss/total_loss‚35.34882274531212 trainer/global_step‚2199 /loss/FRt1/detection_loss‚3.5713377120318235 _runtime‚ 907.945393212¯:×jœ¡‚(:‘ µïÄÓÆõ¸Û gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚152.266' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚84.59222222222222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­:Ñœ¢‚(:‘  µïÄÓôÞëÞ disk.rbd0.in‚0 disk.rbd5.out‚ 0.306640625 disk.dm-1.out‚ 42.7421875 proc.cpu.threads‚29 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 161.53515625 disk.sda3.out‚ 42.7421875 disk.rbd4.out‚ 0.36962890625( proc.memory.availableMB‚ 38527.703125 network.recv‚171223 disk.sda2.out‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.out‚ 3715.81640625 disk.rbd4.in‚0 disk.drbd1743.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚82.37858873409559 disk.rbd2.in‚0" disk.nvme0n1.out‚ 3715.81640625 disk.dm-0.in‚ 161.53515625 disk.dm-0.out‚ 3715.81640625 disk.dm-1.in‚0.109375# proc.memory.rssMB‚ 11266.2265625 cpu‚14.23688694557832 network.sent‚ 2.016975e+06$ disk./.usageGB‚1227.500503540039 disk.drbd1743.out‚ 0.95703125$ memory_percent‚41.21139049530029 disk.nvme0n1.in‚ 161.53515625 disk.rbd5.in‚0 disk.sda.out‚ 42.7421875 disk.rbd2.out‚ 0.306640625 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda3.in‚0.109375 disk.sda.in‚0.109375* proc.memory.percent‚17.190897464752197 disk.sda1.in‚0 disk.rbd1.in‚0`Á”0£+ _wandbruntime‚927  _runtime‚927p´:*ǤÁ $ _timestamp‚1.785804738028701e+09 _runtime‚ 927.301986812 _step‚176 ' lr-AdamW/pg1‚5.7961647405453414e-05 ' lr-AdamW/pg2‚5.7961647405453414e-05 trainer/global_step‚2249ãÂ|^Ì¥Æ trainer/global_step‚2249 $ _timestamp‚1.785804738028701e+09 _runtime‚ 927.301986812 _step‚176 ' lr-AdamW/pg1‚5.7961647405453414e-05 ' lr-AdamW/pg2‚5.7961647405453414e-05°Ðáx0¦+ _wandbruntime‚927  _runtime‚927{@~§³O TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23766395 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 927.306565553 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40609164854784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1979509875712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚38629759448064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14951329 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚38721837134848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38717724 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23756739 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚38721862038528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1979422446080 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚39644510131752 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚39644612974236 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14951327 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23756739 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2033200451072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14951327 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15310409 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14951329 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚38629742408704 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚40609269323776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1979422446080 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8860647 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚38629759448064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _step‚177 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1979509875712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1974663537404 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 trainer/global_step‚2249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40609164854784 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6450001 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚38629742408704 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38708066 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2033069776896 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚37669949436832 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚37669932397472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136y?ÇŸ¾'¨¸O MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚40609269323776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15310648 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚38629742408704 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38708066 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2033069776896 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚37669949436832 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚37669932397472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23766395 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 927.306565553 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40609164854784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1979509875712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚38629759448064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14951329 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚38721837134848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38717724 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23756739 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚38721862038528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1979422446080 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚39644510131752 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚39644612974236 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14951327 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23756739 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2033200451072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14951327 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15310409 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14951329 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚38629742408704 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚40609269323776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1979422446080 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8860647 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚38629759448064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _step‚177 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1979509875712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1974663537404 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 trainer/global_step‚2249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40609164854784 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6450001 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚37686689609152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38734904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _step‚178 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23767262 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚40627211237376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38734904 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1975455207502 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚38739069155328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8864582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7858047383638206e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38725232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2033968851456 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14957972 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40773174965248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚38739044251648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15317459 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15317183 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2034105809920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _runtime‚ 927.637447274 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14957972 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚40627315713536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14957970 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚38646909171712 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚38646909171712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23767262 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚183ñ|xñ«ÂM trainer/global_step‚2249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6452601 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1980302065664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6452875 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚37686689609152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38734904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _step‚178 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23767262 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚40627211237376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38734904 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1975455207502 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚38739069155328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8864582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7858047383638206e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38725232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2033968851456 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14957972 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40773174965248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚38739044251648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15317459 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15317183 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2034105809920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _runtime‚ 927.637447274 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14957972 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.curreÖV›c× nt‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚40627315713536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14957970 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚38646909171712 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚38646909171712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23767262 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23776932 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚37686672569792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14957970 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39662127777294 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680²°  !0¬+ _wandbruntime‚927  _runtime‚927&%â-­û /loss/FRt2/assignment_loss‚3.892594124681206 2loss/SRqqt1/assignment_loss‚1.5915084660518914 0loss/FBt2/assignment_loss‚0.3854743541722366 /loss/FRt2/detection_loss‚3.1033914682845913 _runtime‚ 927.638512591 2loss/SRqqt2/assignment_loss‚1.7167770119456502 .loss/FRt1/detection_loss‚3.841386127713564 1loss/SRqqt1/detection_loss‚1.5260974844303523 1loss/SRqqt2/detection_loss‚2.9487052286962006 % _timestamp‚1.7858047383650293e+09 _step‚179 /loss/FRt1/assignment_loss‚4.108277111440091 trainer/global_step‚2249 1loss/FBt1/assignment_loss‚0.05901974884597751 .loss/FBt1/detection_loss‚3.089302604204689 .loss/FBt2/detection_loss‚6.529410698916788 &loss/total_loss‚32.791944429383236 epoch‚0¨Ad˜®€ 1loss/SRqqt1/detection_loss‚1.5260974844303523 1loss/SRqqt2/detection_loss‚2.9487052286962006 % _timestamp‚1.7858047383650293e+09 _step‚179 /loss/FRt1/assignment_loss‚4.108277111440091 trainer/global_step‚2249 1loss/FBt1/assignment_loss‚0.05901974884597751 .loss/FBt1/detection_loss‚3.089302604204689 .loss/FBt2/detection_loss‚6.529410698916788 &loss/total_loss‚32.791944429383236 epoch‚0 /loss/FRt2/assignment_loss‚3.892594124681206 2loss/SRqqt1/assignment_loss‚1.5915084660518914 0loss/FBt2/assignment_loss‚0.3854743541722366 /loss/FRt2/detection_loss‚3.1033914682845913 _runtime‚ 927.638512591 2loss/SRqqt2/assignment_loss‚1.7167770119456502 .loss/FRt1/detection_loss‚3.841386127713564³0ïOœ¯‚(:‘ ÄïÄÓÚÞËÛ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚143.635' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚79.79722222222222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºÑŒ‘™°‚(:Ž  ÄïÄÓìÔåÞ" disk.dm-0.out‚3716.51513671875 disk.rbd5.out‚ 0.306640625 disk.drbd1743.in‚0# proc.memory.rssMB‚ 11266.2265625 proc.cpu.threads‚29% disk./.usageGB‚1227.2876434326172 disk.dm-1.out‚43.25 disk.rbd1.in‚0 disk.nvme0n1.in‚ 161.53515625 disk.rbd1.out‚ 0.306640625 cpu‚14.234177162409564 disk.dm-0.in‚ 161.53515625 disk.sda.out‚43.25' disk.nvme0c0n1.out‚3716.51513671875 network.recv‚173295 disk.sda3.out‚43.25 disk.rbd0.in‚0 disk.sda.in‚0.109375 disk.dm-1.in‚0.109375 disk.sda1.in‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 161.53515625 disk.sda3.in‚0.109375 disk.rbd4.in‚0 disk.rbd5.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0$ memory_percent‚41.21142625808716) proc.memory.availableMB‚ 38527.6796875* proc.memory.percent‚17.190897464752197 network.sent‚ 2.061714e+06 disk.rbd0.out‚0 disk.rbd4.out‚ 0.36962890625 disk.rbd2.out‚ 0.306640625 disk.sda2.out‚0% disk.nvme0n1.out‚3716.51513671875) disk./.usagePercent‚82.36430351368497 disk.drbd1743.out‚ 0.96484375 disk.rbd2.in‚0 disk.rbd3.out‚0}s‡Kœ±‚(:‘ ÓïÄÓµÌìÛ gpu.0.gpu‚42.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚119.721' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚66.51166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í徉©²‚(:ž  ÓïÄÓò†¥ß disk.sda.out‚ 44.08984375 disk.rbd4.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0& disk.nvme0c0n1.out‚3779.9580078125$ disk.nvme0n1.out‚3779.9580078125 disk.sda3.in‚0.109375 cpu‚14.231887213080862 network.sent‚ 2.106471e+06$ disk./.usageGB‚1227.388656616211! disk.dm-0.out‚3779.9580078125 disk.dm-1.out‚ 44.08984375) disk./.usagePercent‚82.37108259319166 disk.dm-0.in‚ 161.53515625 disk.rbd5.out‚ 0.306640625 disk.drbd1743.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 11266.2265625* proc.memory.percent‚17.190897464752197 disk.rbd0.out‚0 disk.sda.in‚0.109375 disk.sda3.out‚ 44.08984375 disk.rbd2.out‚ 0.306640625" disk.nvme0c0n1.in‚ 161.53515625* proc.memory.availableMB‚38527.26171875 network.recv‚175849 disk.sda2.out‚0% memory_percent‚41.212064027786255 disk.rbd5.in‚0 disk.drbd1743.out‚ 0.97265625 disk.rbd1.in‚0 disk.rbd1.out‚ 0.306640625 disk.nvme0n1.in‚ 161.53515625 proc.cpu.threads‚29 disk.rbd0.in‚0 disk.rbd4.out‚ 0.36962890625 disk.dm-1.in‚0.109375 disk.sda1.in‚0 disk.sda1.out‚0èâ-J0³+ _wandbruntime‚946  _runtime‚946Ë.vÅÈ´Â _step‚180 ' lr-AdamW/pg1‚5.9250256729718725e-05 ' lr-AdamW/pg2‚5.9250256729718725e-05 trainer/global_step‚2299 % _timestamp‚1.7858047573904593e+09 _runtime‚ 946.663735204mº„Q굂 ' lr-AdamW/pg1‚5.9250256729718725e-05 ' lr-AdamW/pg2‚5.9250256729718725e-05 trainer/global_step‚2299 % _timestamp‚1.7858047573904593e+09 _runtime‚ 946.663735204 _step‚180´XÂÁ]0¶+ _wandbruntime‚946  _runtime‚946僷'·±O LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15651795 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2023418581504 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39577876 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚38506941013472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2018466197788 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚39488097598464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2078414438400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚40525510053744 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41511499140608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2023506011136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 $ _timestamp‚1.785804757391402e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 946.66856717 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41511603609600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39568218 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚39582217878528 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24294397 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15283479 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚39488097598464 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15652019 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39582192974848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15283477 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6594522 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40525407211260 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2018552000912 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2023506011136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41511499140608Ý—r`¼'¸¶O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚40525510053744 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41511499140608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2023506011136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 $ _timestamp‚1.785804757391402e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 946.66856717 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41511603609600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39568218 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚39582217878528 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24294397 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15283479 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚39488097598464 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15652019 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39582192974848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15283477 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6594522 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40525407211260 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2018552000912 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2023506011136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41511499140608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15651795 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2023418581504 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39577876 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚38506941013472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2018466197788 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚39488097598464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2078414438400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130µFÅ0¹+ _wandbruntime‚947  _runtime‚947 =Épº½M J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39595056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15290120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39505264361472 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _step‚182 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39505247322112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2024298201088 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24304934 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2079329611264 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39599424995328 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15658547 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 _runtime‚ 947.001698083 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 F6DeviceStatsMonitoÐ9éSr.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15290122 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2024298201088 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40543024856802 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39585384 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39505247322112 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15658795 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2019429474246 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39505264361472 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9061432 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38523681185792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41529649999360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9061434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚2299 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚41529649999360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39595056 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6597361 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39585384 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39599400091648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚38523698225152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2019343671010 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40543127699398 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39599400091648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚38523698225152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2019343671010 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40543127699398 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39595056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15290120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39505264361472 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _step‚182 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39505247322112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2024298201088 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24304934 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2079329611264 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39599424995328 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15658547 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 _runtime‚ 947.001698083 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15290122 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2024298201088 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40543024856802 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39585384 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39505247322112 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15658795 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2019429474246 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39505264361472 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9061432 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38523681185792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41529649999360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9061434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚2299 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚41529649999360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39595056 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6597361¶ö3üÒ0¼+ _wandbruntime‚947  _runtime‚947‘(þ½ø % _timestamp‚1.7858047577292986e+09 _runtime‚ 947.002482152 /loss/FRt1/assignment_loss‚4.051949055989583 /loss/FRt2/assignment_loss‚4.363060611359616 /loss/FRt1/detection_loss‚3.7689313407739005 trainer/global_step‚2299 _step‚183 epoch‚0 2loss/SRqqt1/assignment_loss‚1.4008068948768186 1loss/SRqqt2/assignment_loss‚1.946022475596334 1loss/SRqqt1/detection_loss‚1.3715550067565014 0loss/FBt2/assignment_loss‚0.1739336377796917 #loss/total_loss‚41.725180750424 1loss/FBt1/assignment_loss‚0.06781042870352394 /loss/FRt2/detection_loss‚3.0773058780964386 0loss/SRqqt2/detection_loss‚3.634060535641776 .loss/FBt1/detection_loss‚7.549294376435383 /loss/FBt2/detection_loss‚10.320450508414433ìÛ:í¾ý epoch‚0 2loss/SRqqt1/assignment_loss‚1.4008068948768186 1loss/SRqqt2/assignment_loss‚1.946022475596334 1loss/SRqqt1/detection_loss‚1.3715550067565014 0loss/FBt2/assignment_loss‚0.1739336377796917 #loss/total_loss‚41.725180750424 1loss/FBt1/assignment_loss‚0.06781042870352394 /loss/FRt2/detection_loss‚3.0773058780964386 0loss/SRqqt2/detection_loss‚3.634060535641776 .loss/FBt1/detection_loss‚7.549294376435383 /loss/FBt2/detection_loss‚10.320450508414433 % _timestamp‚1.7858047577292986e+09 _runtime‚ 947.002482152 /loss/FRt1/assignment_loss‚4.051949055989583 /loss/FRt2/assignment_loss‚4.363060611359616 /loss/FRt1/detection_loss‚3.7689313407739005 trainer/global_step‚2299 _step‚183·à »Pœ¿‚(:‘ âïÄÓŽ÷âÛ gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚148.188' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚82.32666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JÑJñÀ‚(:’  âïÄÓ¶ºÞ% disk./.usageGB‚1227.3885536193848 disk.sda.out‚ 44.5390625 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 0.306640625* proc.memory.availableMB‚38527.46484375 disk.rbd5.out‚ 0.306640625 disk.sda.in‚0.109375 disk.rbd4.out‚ 0.36962890625$ memory_percent‚41.21175408363342 cpu‚14.229698971509691 disk.sda3.out‚ 44.5390625 disk.rbd0.out‚0 disk.rbd5.in‚0 disk.rbd2.out‚ 0.306640625 disk.rbd1.in‚0) disk./.usagePercent‚82.37107568098824 proc.cpu.threads‚29 disk.sda3.in‚0.109375 disk.rbd4.in‚0 disk.drbd1743.in‚0 disk.drbd1743.out‚1# proc.memory.rssMB‚ 11266.2265625* proc.memory.percent‚17.190897464752197 network.sent‚ 2.108963e+06 disk.rbd0.in‚0! disk.dm-0.out‚3780.1845703125 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 161.53515625 disk.nvme0n1.in‚ 161.53515625 network.recv‚176934 disk.dm-0.in‚ 161.53515625 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 44.5390625 disk.sda1.in‚0 disk.rbd3.in‚0& disk.nvme0c0n1.out‚3780.1845703125$ disk.nvme0n1.out‚3780.1845703125G¿ß‡0Á+ _wandbruntime‚966  _runtime‚966¤ˆuÆÂÀ & lr-AdamW/pg1‚6.053886605398402e-05 & lr-AdamW/pg2‚6.053886605398402e-05 trainer/global_step‚2349 % _timestamp‚1.7858047768566382e+09 _runtime‚ 966.129939485 _step‚184Ï„ {ËÃÅ & lr-AdamW/pg2‚6.053886605398402e-05 trainer/global_step‚2349 % _timestamp‚1.7858047768566382e+09 _runtime‚ 966.129939485 _step‚184 & lr-AdamW/pg1‚6.053886605398402e-05¸÷Ÿ30Ä+ _wandbruntime‚966  _runtime‚966Sôé ¹'ųO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42413833426432 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40438028 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15615629 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15990740 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40442548814848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2062354661296 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24822399 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40346418709504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15615627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24812743 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9254347 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚39343966668832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24822399 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 % _timestamp‚1.7858047768576055e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15615627 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9254345 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚42566280876544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6736156 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚42566129492992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2067502146560 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40442573718528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚41406407133252 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2062440464420 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2349 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24822399 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40346418709504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15615627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24812743 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9254347 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚39343966668832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24822399 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 % _timestamp‚1.7858047768576055e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15615627 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9254345 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚42566280876544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6736156 \ODeviceStatsMonitor.on_train_batch_sÕÂÆiHtart/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚42566129492992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2067502146560 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40442573718528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚41406407133252 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2062440464420 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2349 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42413833426432 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40438028 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15615629 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15990740 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40442548814848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2062354661296 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚41424024778906 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 trainer/global_step‚2349 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40455208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2068294336512 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40363602511872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15622270 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15997287 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚39360706841152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2063317937754 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42431984285184 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚40363585472512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15997542 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2124488378880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 966.465773393 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24832936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6739258 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42431879809024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2124621143040 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚42584401978368 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39360689801792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40455208 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39360689801792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40455208 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚41424024778906 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 trainer/global_step‚2349 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40455208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2068294336512 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40363602511872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15622270 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15997287 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚39360706841152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2063317937754 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42431984285184 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚40363585472512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15997542 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2124488378880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 966.465773393 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24832936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6739258 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42431879809024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2124621143040 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚42584401978368ºBõ~Á0Ê+ _wandbruntime‚966  _runtime‚966­pcOËü /loss/FBt1/detection_loss‚5.6353687843011055 %loss/total_loss‚42.83666821652733 /loss/FRt1/assignment_loss‚4.433215728620204 /loss/FRt2/assignment_loss‚4.315879788061585 2loss/SRqqt1/assignment_loss‚1.4598468305861079 2loss/SRqqt2/assignment_loss‚1.8446342823259971 .loss/FRt1/detection_loss‚3.392682029343233 trainer/global_step‚2349 /loss/FRt2/detection_loss‚2.8591087964749096 _runtime‚ 966.466502549 0loss/SRqqt2/detection_loss‚2.619077403895812 epoch‚0 2loss/FBt1/assignment_loss‚0.003970806962837849 1loss/FBt2/assignment_loss‚0.48667819367528864 1loss/SRqqt1/detection_loss‚1.4832322346051576 % _timestamp‚1.7858047771933951e+09 .loss/FBt2/detection_loss‚14.30297333767509 _step‚187*ÔÌ epoch‚0 2loss/FBt1/assignment_loss‚0.003970806962837849 1loss/FBt2/assignment_loss‚0.48667819367528864 1loss/SRqqt1/detection_loss‚1.4832322346051576 % _timestamp‚1.7858047771933951e+09 .loss/FBt2/detection_loss‚14.30297333767509 _step‚187 /loss/FBt1/detection_loss‚5.6353687843011055 %loss/total_loss‚42.83666821652733 /loss/FRt1/assignment_loss‚4.433215728620204 /loss/FRt2/assignment_loss‚4.315879788061585 2loss/SRqqt1/assignment_loss‚1.4598468305861079 2loss/SRqqt2/assignment_loss‚1.8446342823259971 .loss/FRt1/detection_loss‚3.392682029343233 trainer/global_step‚2349 /loss/FRt2/detection_loss‚2.8591087964749096 _runtime‚ 966.466502549 0loss/SRqqt2/detection_loss‚2.619077403895812»_¯gƒ›Í‚(: ñïÄÓ¨±ËÛ gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚154.45' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚85.80555555555554 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘ڧ΂(:œ  ñïÄÓèŽÌÞ disk.rbd4.in‚0 disk.dm-1.in‚0.109375 disk.drbd1743.in‚0 disk.sda2.in‚0% disk.nvme0n1.out‚3780.67041015625 proc.cpu.threads‚29) disk./.usagePercent‚82.37107977710878 disk.sda.in‚0.109375 disk.rbd5.out‚ 0.306640625 disk.dm-1.out‚ 45.4140625 disk.rbd2.out‚ 0.306640625 disk.sda1.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 161.53515625* proc.memory.percent‚17.190897464752197 disk.rbd2.in‚0 disk.rbd0.in‚0$ disk./.usageGB‚1227.388614654541 disk.sda.out‚ 45.4140625 disk.rbd4.out‚ 0.36962890625 disk.drbd1743.out‚ 1.01953125 disk.nvme0n1.in‚ 161.53515625 disk.sda3.in‚0.109375$ memory_percent‚41.21206998825073 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.306640625) proc.memory.availableMB‚ 38527.2578125" disk.dm-0.out‚3780.67041015625 disk.sda2.out‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 161.53515625 disk.rbd0.out‚0 disk.rbd5.in‚0' disk.nvme0c0n1.out‚3780.67041015625 disk.sda3.out‚ 45.4140625# proc.memory.rssMB‚ 11266.2265625 cpu‚14.227428908775817 network.sent‚ 2.153858e+06 network.recv‚179386d¶k0Ï+ _wandbruntime‚985  _runtime‚985±¼ÆÐÀ & lr-AdamW/pg1‚6.182747537824933e-05 & lr-AdamW/pg2‚6.182747537824933e-05 trainer/global_step‚2399 % _timestamp‚1.7858047961945267e+09 _runtime‚ 985.467840485 _step‚188‘Ó[ËÑÅ & lr-AdamW/pg2‚6.182747537824933e-05 trainer/global_step‚2399 % _timestamp‚1.7858047961945267e+09 _runtime‚ 985.467840485 _step‚188 & lr-AdamW/pg1‚6.182747537824933e-05¼ºU 0Ò+ _wandbruntime‚985  _runtime‚985Óz)‚/ Ó³O YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43471688507904 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41302929558528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚40180975284832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6880749 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚43316167712256 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ?9DeviceStatsMàÿÀŠonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16331946 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41302904654848 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2106243124804 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2111498281984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚42287201370276 trainer/global_step‚2399 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15947777 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2168914527232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚42287304212760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚43316272181248 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43471844085760 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚41204773899264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚189 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2111410852352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25350401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚41204773899264 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130¦¯s4¾'Ô¸O [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚41204756859904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25340745 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15947779 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚40180958245472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2106328927928 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2168914527232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚42287304212760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚43316272181248 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43471844085760 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚41204773899264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚189 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2111410852352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25350401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚41204773899264 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43471688507904 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41302929558528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚40180975284832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6880749 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚43316167712256 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16331946 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41302904654848 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2106243124804 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2111498281984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚42287201370276 trainer/global_step‚2399 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15947777 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25351268 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2169682788352 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚42304819015818 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚190 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41320111771648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2107206401262 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16338502 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚43334214094848 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41320136675328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2169815552512 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43489952227840 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41315360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6883370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚41221940662272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚42304921858414 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1íͻ̢×ÂM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25351268 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2169682788352 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚42304819015818 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚190 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41320111771648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2107206401262 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16338502 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚43334214094848 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41320136675328 ZMDeviceStatsMonitoÊiQ&r.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2169815552512 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43489952227840 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41315360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6883370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚41221940662272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚42304921858414 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16338753 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9455134 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.785804815557363e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16279929 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚42163260494848 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42063112049664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2213989336064 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41017983900832 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 _runtime‚1004.834555603 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43168201292268 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25878403 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚43168098449784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44218606467072 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9648045 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44218501998080 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚42163285398528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16672185 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚42163260494848 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42063112049664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2213989336064 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41017983900832 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 _runtime‚1004.834555603 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43168201292268 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25878403 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚43168098449784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44218606467072 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9648045 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44218501998080 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚42163285398528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16672185 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.785804815557363e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16279929 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚43185716095326 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42080261773312 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25888940 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.p&:¬eak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16286570 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7858048158959153e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚44236652856832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41034707033792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚194 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2156374044160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16286570 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42080278812672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16678727 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2156374044160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚44395358924288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚2449 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9651982 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44236548380672 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚42180492515328 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2214891312640 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42175512 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16286572 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44236652856832 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2151094864770 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25879270 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16286572 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7027004 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25888940 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42175512 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7026745 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2qÝ­É&çÃM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42080278812672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42175512 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16286572 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44236652856832 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2151094864770 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25879270 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16286572 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7027004 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25888940 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42175512 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7026745 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚43185716095326 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42080261773312 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25888940 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16286570 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7858048158959153e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚44236652856832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41034707033792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚194 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2156374044160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16286570 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42080278812672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16678727 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2156374044160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚44395358924288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚2449 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9651982 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44236548380672 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚42180492515328 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2214891312640 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16612077 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42921450200064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26406405 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 $ _timestamp‚1.785804836930103e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43008826 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45282962191872 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚45282802419712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43023641238528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17011936 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43023616334848 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9844897 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26396749 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚1026.207613127 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44049098371776 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2194105854944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45120836283904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42921433160704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2259320953344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2199490552832 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2199490552832 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2199403123200 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17011669 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16612079 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42921433160704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16612079 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26396749 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7167039 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44048995529292 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2194020051820 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9844895 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2259186084864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42921450200064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 trainer/global_step‚2499 "ZOò¸O PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26406405 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43018484 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 _step‚197 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2194105854944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45120836283904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42921433160704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2259320953344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2199490552832 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2199490552832 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2199403123200 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17011669 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16612079 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42921433160704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16612079 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26396749 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7167039 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44048995529292 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2194020051820 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9844895 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2259186084864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42921450200064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 trainer/global_step‚2499 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45120836283904 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41854975477472 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16612077 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42921450200064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26406405 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 $ _timestamp‚1.785804836930103e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43008826 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45282962191872 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚45282802419712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43023641238528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17011936 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43023616334848 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9844897 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26396749 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚1026.207613127 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44049098371776 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43035664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2200282742784 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7169942 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 % _timestamp‚1.7858048373295553e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42938616963072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43025992 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45300908471808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 _runtime‚1026.603292228 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26416942 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42938599923712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17018515 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42938616963072 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2200370179584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2260221978624 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44066716017430 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2194983328278 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚198 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43025992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45301070333952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41871715649792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16618720 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43040823451648 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2200282742784 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45138987142656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16618722 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2260085020160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41871732689152 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 trainer/global_step‚2499 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9848834 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7169683 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251^¬‹ØÉ&õÃM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2194897525042 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16618722 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9848832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43040848355328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43035664 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26407272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42938599923712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26407272 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44066613174834 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2200370179584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43035664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2200282742784 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7169942 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 % _timestamp‚1.7858048373295553e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42938616963072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43025992 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45300908471808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 _runtime‚1026.603292228 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26416942 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42938599923712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17018515 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42938616963072 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2200370179584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2260221978624 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44066716017430 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2194983328278 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚198 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43025992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45301070333952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41871715649792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16618720 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43040823451648 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2200282742784 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45138987142656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16618722 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2260085020160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41871732689152 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 trainer/global_step‚2499 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9848834 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7169683 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251Æ7Äf‚2ö- _wandbruntime‚1026  _runtime‚1026͈¾Eý÷÷ $ _timestamp‚1.785804837331028e+09 _step‚199 1loss/SRqqt1/detection_loss‚1.4447395007579824 /loss/SRqqt2/detection_loss‚3.99159403230587 .loss/FBt1/detection_loss‚4.926133128307955 .loss/FBt2/detection_loss‚6.349975866679055 trainer/global_step‚2499 2loss/SRqqt1/assignment_loss‚1.3874777064743367 0loss/FBt1/assignment_loss‚0.3680434737287119 .loss/FRt2/detection_loss‚3.106411695141684 _runtime‚1026.604188699 /loss/FRt2/assignment_loss‚3.815229270945896 1loss/SRqqt2/assignment_loss‚2.053559799646509 %loss/total_loss‚35.44546499538735 epoch‚0 /loss/FRt1/assignment_loss‚3.904590755701065 1loss/FBt2/assignment_loss‚0.21269010307263497 /loss/FRt1/detection_loss‚3.8850196626256492Ö Eøü 2loss/SRqqt1/assignment_loss‚1.3874777064743367 0loss/FBt1/assignment_loss‚0.3680434737287119 .loss/FRt2/detection_loss‚3.106411695141684 _runtime‚1026.604188699 /loss/FRt2/assignment_loss‚3.815229270945896 1loss/SRqqt2/assignment_loss‚2.053559799646509 %loss/total_loss‚35.44546499538735 epoch‚0 /loss/FRt1/assignment_loss‚3.904590755701065 1loss/FBt2/assignment_loss‚0.21269010307263497 /loss/FRt1/detection_loss‚3.8850196626256492 $ _timestamp‚1.785804837331028e+09 _step‚199 1loss/SRqqt1/detection_loss‚1.4447395007579824 /loss/SRqqt2/detection_loss‚3.99159403230587 .loss/FBt1/detection_loss‚4.926133128307955 .loss/FBt2/detection_loss‚6.349975866679055 trainer/global_step‚2499ǽ§‘Lœù‚(:‘ ­ðÄÓ´ÄÝÛ gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚142.568' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚79.20444444444445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â–f%¡ú‚(:–  ­ðÄÓ̧âÞ disk.dm-1.in‚0.109375 disk.sda2.in‚0 disk.rbd5.in‚0 disk.rbd2.out‚ 0.306640625& disk.nvme0c0n1.out‚3907.6025390625 disk.dm-0.in‚ 161.53515625# proc.memory.rssMB‚ 11266.2265625 network.sent‚ 2.311842e+06 network.recv‚187901 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0! disk.dm-0.out‚3907.6025390625 proc.cpu.threads‚29% disk./.usageGB‚1227.4701843261719 disk.rbd4.out‚ 0.36962890625 disk.drbd1743.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda.in‚0.109375 disk.sda.out‚ 48.265625 disk.rbd5.out‚ 0.306640625 disk.nvme0n1.in‚ 161.53515625 disk.sda3.in‚0.109375) proc.memory.availableMB‚ 38527.9140625) disk./.usagePercent‚82.37655398621344 disk.rbd4.in‚0 disk.drbd1743.out‚1.078125 disk.rbd2.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 161.53515625$ disk.nvme0n1.out‚3907.6025390625 disk.sda2.out‚0 disk.rbd1.out‚ 0.306640625 disk.sda3.out‚ 48.265625% memory_percent‚41.211068630218506* proc.memory.percent‚17.190897464752197 disk.dm-1.out‚ 48.265625 cpu‚14.218517959308006•ô ¢2û- _wandbruntime‚1048  _runtime‚1048ˆ÷µüÁ & lr-AdamW/pg2‚6.569330335104525e-05 trainer/global_step‚2549 % _timestamp‚1.7858048589072745e+09 _runtime‚1048.180641444 _step‚2tØÂ¥*00 & lr-AdamW/pg1‚6.569330335104525e-05})J<ÌýÆ & lr-AdamW/pg1‚6.569330335104525e-05 & lr-AdamW/pg2‚6.569330335104525e-05 trainer/global_step‚2549 % _timestamp‚1.7858048589072745e+09 _runtime‚1048.180641444 _step‚200È m¦}2þ- _wandbruntime‚1048  _runtime‚1048eB²º¼'ÿ¶O XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚43779771311104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2243486688256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚42692001132832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2243486688256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚42691984093472 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2304507836928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44929995451284 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 trainer/global_step‚2549 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7309787 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46188347240448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46023170569728 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚43779788350464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17351266 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46023275038720 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43878636 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚42691984093472 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2304507836928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44929995451284 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 trainer/global_step‚2549 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7309787 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46188347240448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46023170569728 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚43779788350464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17351266 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46023275038720 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43878636 I=2- _wandbruntime‚1048  _runtime‚1048ù®EÆ&‚ÀM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚42708724265792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚43796955113472 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43886144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43895816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10045684 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44947510254342 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17357991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚43796955113472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26935274 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚46041321428480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2238785988550 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26935274 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2244278878208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43895816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16950870 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2305408878592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _runtime‚1048.580808864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2244366315008 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26944944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚46041321428480 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43796938074112 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26944944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43796938074112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7312552ÿD +ƒÅM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43886144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 _step‚202 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46206613073920 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7312552 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚42708724265792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚43796955113472 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43886144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43895816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10045684 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44947510254342 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17357991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚43796955113472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26935274 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚46041321428480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2238785988550 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26935274 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2244278878208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43895816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16950870 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2305408878592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _runtime‚1048.580808864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2244366315008 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26944944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚46041321428480 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43796938074112 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26944944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43796938074112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46925609324544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17276379 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43528992709472 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 % _timestamp‚1.7858048803973882e+09 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚45810892530792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚47094050530816 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚47093897050112 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 _runtime‚1069.674547208 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2287395394048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44638109461504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27462409 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚44744328014848 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46925609324544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17276377 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7452388 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27452753 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2349569035264 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17276377 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17691226 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43529009748832 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚44744352918528 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46925504855552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46925504855552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17690983 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44638109461504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚45810789688308 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46925609324544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17276379 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43528992709472 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 % _timestamp‚1.7858048803973882e+09 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚45810892530792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚47094050530816 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚47093897050112 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 _runtime‚1069.674547208 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2287395394048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44638109461504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27462409 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚44744328014848 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46925609324544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17276377 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7452388 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27452753 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2349569035264 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17276377 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17691226 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43529009748832 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚44744352918528 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46925504855552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46925504855552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17690983 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44638109461504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚45810789688308 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17283022 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44746296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2350598727680 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚45828407333850 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44755968 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44746296 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 % _timestamp‚1.7858048807361946e+09 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43545749921152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17697990 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17697754 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44655276224512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚46943551238144 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44755968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚44761535131648 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 trainer/global_step‚2599 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44655293263872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10242534 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27463276 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27463276 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚46943655714304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚206 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27472946 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2288362450432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2282674452058 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7455456 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚46943655714304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44655276224512 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7455222 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17283022 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17283020 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚47112158763008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43545732881792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17283020 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚47112007386624 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2288275013632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚45828510176446 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10242532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17697754 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44655276224512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚46943551238144 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44755968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚44761535131648 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 trainer/global_step‚2599 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44655293263872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10242534 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27463276 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27463276 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚46943655714304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚206 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27472946 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2288362450432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2282674452058 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7455456 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚46943655714304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44655276224512 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7455222 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17283022 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17283020 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚47112158763008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43545732881792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17283020 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚47112007386624 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2288275013632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚45828510176446 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10242532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17283022 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44746296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2350598727680 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚45828407333850 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44755968 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44746296 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 % _timestamp‚1.7858048807361946e+09 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43545749921152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17697990Îéô:¹2”- _wandbruntime‚1070  _runtime‚1070/šD8ÿ•ù .loss/FBt1/detection_loss‚4.104174373580607 1loss/SRqqt1/assignment_loss‚1.502234660665549 1loss/FBt1/assignment_loss‚0.21380914485234825 1loss/FBt2/assignment_loss‚0.32421060575372257 .loss/FRt1/detection_loss‚3.117460400575683 -loss/FRt2/detection_loss‚3.22588371347498 _runtime‚1070.010497311 /loss/FRt1/assignment_loss‚4.121590080715361 /loss/FRt2/assignment_loss‚3.777847363625044 epoch‚0 trainer/global_step‚2599 % _timestamp‚1.7858048807373788e+09 .loss/FBt2/detection_loss‚5.085731306255588 &loss/total_loss‚31.793790016312748 2loss/SRqqt2/assignment_loss‚1.7838456028364074 1loss/SRqqt1/detection_loss‚1.7600865646245423 0loss/SRqqt2/detection_loss‚2.776916199352918 _step‚207jק–þ .loss/FBt1/detection_loss‚4.104174373580607 1loss/SRqqt1/assignment_loss‚1.502234660665549 1loss/FBt1/assignment_loss‚0.21380914485234825 1loss/FBt2/assignment_loss‚0.32421060575372257 .loss/FRt1/detection_loss‚3.117460400575683 -loss/FRt2/detection_loss‚3.22588371347498 _runtime‚1070.010497311 /loss/FRt1/assignment_loss‚4.121590080715361 /loss/FRt2/assignment_loss‚3.777847363625044 epoch‚0 trainer/global_step‚2599 % _timestamp‚1.7858048807373788e+09 .loss/FBt2/detection_loss‚5.085731306255588 &loss/total_loss‚31.793790016312748 2loss/SRqqt2/assignment_loss‚1.7838456028364074 1loss/SRqqt1/detection_loss‚1.7600865646245423 0loss/SRqqt2/detection_loss‚2.776916199352918 _step‚207ÏCp=—‚(:… ÚðÄÓÊíÀÛ gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚123.894' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚68.83 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ.Æ> ˜‚(:•  ÚðÄÓ”šß disk.rbd5.in‚0 disk.dm-1.in‚0.109375 disk.sda1.in‚0 disk.rbd1.in‚0* proc.memory.availableMB‚38527.13671875* proc.memory.percent‚17.190897464752197 disk.rbd3.out‚0# proc.memory.rssMB‚ 11266.2265625) disk./.usagePercent‚82.37654937807783 disk.rbd0.in‚0 disk.drbd1743.out‚1.140625 disk.rbd2.in‚0 disk.sda2.in‚0 network.recv‚193709 disk.sda.out‚ 50.16796875 disk.rbd2.out‚ 0.4873046875" disk.nvme0c0n1.in‚ 161.53515625 disk.rbd1.out‚ 0.455078125 cpu‚14.21115470703844 proc.cpu.threads‚29 disk.rbd0.out‚0 disk.sda2.out‚0" disk.nvme0n1.out‚ 3908.59765625 disk.sda3.out‚ 50.16796875$ disk./.usageGB‚1227.470115661621 disk.dm-0.in‚ 161.53515625 disk.dm-0.out‚ 3908.59765625 disk.dm-1.out‚ 50.16796875 disk.drbd1743.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 161.53515625% memory_percent‚41.212254762649536 network.sent‚ 2.40406e+06 disk.rbd5.out‚ 0.306640625 disk.sda1.out‚0$ disk.nvme0c0n1.out‚ 3908.59765625 disk.sda.in‚0.109375 disk.rbd4.in‚0 disk.rbd4.out‚ 0.36962890625 disk.sda3.in‚0.109375 ÍûŸ2™- _wandbruntime‚1088  _runtime‚1088Û¤‡ÇšÁ & lr-AdamW/pg1‚6.827052199957586e-05 & lr-AdamW/pg2‚6.827052199957586e-05 trainer/global_step‚2649 % _timestamp‚1.7858048995377843e+09 _runtime‚1088.811055742 _step‚208éÝÖäÌ›Æ _step‚208 & lr-AdamW/pg1‚6.827052199957586e-05 & lr-AdamW/pg2‚6.827052199957586e-05 trainer/global_step‚2649 % _timestamp‚1.7858048995377843e+09 _runtime‚1088.811055742ЖTP@2œ- _wandbruntime‚1088  _runtime‚1088<¢v;Y µO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45496447611904 _step‚209 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45604683854848 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 trainer/global_step‚2649 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2331478959104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45604708758528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18030379 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27990411 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚47827839141376 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2325771245468 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2325685442344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _runtime‚1088.815557279 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45589282 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17608527 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45496464651264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45496447611904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18030623 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2331391529472 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚47827839141376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27990411 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7595176 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2394890732032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10435445 \ODeviceg†6¬bStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚44366018364832 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚47999448107008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚47827943610368 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45598940 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27980755 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7594934 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚44366001325472 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18¬~~À'žºO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚47999448107008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚47827943610368 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45598940 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27980755 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7594934 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚44366001325472 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45496447611904 _step‚209 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45604683854848 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 trainer/global_step‚2649 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2331478959104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45604708758528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18030379 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27990411 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚47827839141376 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2325771245468 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2325685442344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _runtime‚1088.815557279 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45589282 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17608527 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45496464651264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45496447611904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18030623 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2331391529472 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚47827839141376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27990411 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7595176 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2394890732032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10435445 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚44366018364832Ñ2(©2Ÿ- _wandbruntime‚1089  _runtime‚1089­Õ|Æ& ÀM ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7597701 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚44382741497792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2326648718802 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7858048998760636e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 _step‚210 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚47845885523968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚44382758537152 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17615172 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2326562915566 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚47845990000128 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45513631414272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45621890971648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45606448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18037083 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _runtime‚1089.149645603 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚45513631414272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2332358585856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27991278 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2332358585856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18037327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10439384 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17615170 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48017718978560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45616120 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27991278 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7597701 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚44382741497792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2326648718802 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7858048998760636e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 _step‚210 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚47845885523968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚44382758537152 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17615172 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2326562915566 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚47845990000128 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45513631414272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45621890971648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45606448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18037083 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _runtime‚1089.149645603 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚45513631414272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2332358585856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27991278 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2332358585856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18037327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10439384 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17615170 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48017718978560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45616120 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27991278 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚46354785762304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17940679 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18370780 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 % _timestamp‚1.7858049191463075e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2375475094528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48730173427200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2375387664896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7738238 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48730173427200 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚46354802801664 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46459092 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18370533 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46465039694848 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 _step‚213 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10632295 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2375387664896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17940677 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚48730277896192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45203026980832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10632297 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2439944290816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28518413 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46465064598528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28508757 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48904983985664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28508757 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47572686689808 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2369659708976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2375475094528 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46449434 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚2699 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48730277896192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2369573905852J8”Á'¬»O YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 _step‚213 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10632295 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2375387664896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17940677 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚48730277896192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45203026980832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10632297 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2439944290816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28518413 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46465064598528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28508757 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48904983985664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28508757 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47572686689808 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2369659708976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2375475094528 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46449434 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚2699 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48730277896192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2369573905852 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚46354802801664 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚46354785762304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17940679 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18370780 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 % _timestamp‚1.7858049191463075e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2375475094528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48730173427200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2375387664896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7738238 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48730173427200 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚46354802801664 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46459092 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18370533 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46465039694848ÕÊ”xF2­- _wandbruntime‚1108  _runtime‚1108qw]Áo$®ÀM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚46371969564672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10636232 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚46371969564672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2376267284480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 _step‚214 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168153600 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46466600 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48748324285952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28519280 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚46482246811648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48923250769408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17947322 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28528950 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48748219809792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28528950 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47590304335462 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47590201492866 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46476272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18377220 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46476272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1108.754278871 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚48748324285952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46466600 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2376354721280 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2376267284480 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚46371952525312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2370451379074 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚46371952525312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor®ËPÔW.on_train_batch_end/allocation.small_pool.freed‚28519280 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17947320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17947322 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7740988 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0•Î“Ë&¯ÅM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚46371952525312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28519280 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17947320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17947322 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7740988 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚46371969564672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10636232 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚46371969564672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2376267284480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 _step‚214 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168153600 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46466600 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48748324285952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28519280 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚46482246811648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48923250769408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17947322 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28528950 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48748219809792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28528950 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47590304335462 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47590201492866 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46476272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18377220 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46476272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1108.754278871 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚48748324285952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46466600 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2376354721280 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2376267284480 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚46371952525312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2370451379074 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0Ö5¼‹ 2°- _wandbruntime‚1108  _runtime‚1108”Ûw÷ÿ±ù 1loss/SRqqt1/detection_loss‚1.6206154698084028 .loss/FBt1/detection_loss‚5.980439791961601 %loss/total_loss‚32.57064872206315 _step‚215 .loss/FRt2/detection_loss‚3.026706568364586 .loss/FBt2/detection_loss‚4.369058893447812 trainer/global_step‚2699 # _timestamp‚1.78580491948194e+09 0loss/FRt1/assignment_loss‚4.0484238342500065 0loss/FRt2/assignment_loss‚3.7502420984563374 2loss/SRqqt2/assignment_loss‚1.6155272018360465 0loss/FBt2/assignment_loss‚0.1545458011305224 _runtime‚1108.755299544 2loss/SRqqt1/assignment_loss‚1.5424275495862776 /loss/FRt1/detection_loss‚3.5040547453181845 0loss/SRqqt2/detection_loss‚2.419197909337224 epoch‚0 0loss/FBt1/assignment_loss‚0.5394088585661458Ñ|²þ 0loss/FRt1/assignment_loss‚4.0484238342500065 0loss/FRt2/assignment_loss‚3.7502420984563374 2loss/SRqqt2/assignment_loss‚1.6155272018360465 0loss/FBt2/assignment_loss‚0.1545458011305224 _runtime‚1108.755299544 2loss/SRqqt1/assignment_loss‚1.5424275495862776 /loss/FRt1/detection_loss‚3.5040547453181845 0loss/SRqqt2/detection_loss‚2.419197909337224 epoch‚0 0loss/FBt1/assignment_loss‚0.5394088585661458 1loss/SRqqt1/detection_loss‚1.6206154698084028 .loss/FBt1/detection_loss‚5.980439791961601 %loss/total_loss‚32.57064872206315 _step‚215 .loss/FRt2/detection_loss‚3.026706568364586 .loss/FBt2/detection_loss‚4.369058893447812 trainer/global_step‚2699 # _timestamp‚1.78580491948194e+09×DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚48453583769316 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49632507713024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2419471229952 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚49810478949376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10829145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2485083414528 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚48453480926832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2413462369360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2419471229952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚49632507713024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47213123912704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18272827 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29046415 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47319244 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29036759 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47213140952064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚47325395534848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0Ö²<ļ»O _runtime‚1127.798543001 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚48453583769316 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49632507713024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2419471229952 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚49810478949376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10829145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 BDeviceStatsMoni©w•Á tor.on_train_batch_start/allocation.all.current‚9658 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2485083414528 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚48453480926832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2413462369360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2419471229952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚49632507713024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47213123912704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18272827 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29046415 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47319244 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29036759 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47213140952064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚47325395534848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚46040018557472 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130ÙH+52½- _wandbruntime‚1128  _runtime‚1128‘„Å&¾¿M ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚47230307715072 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18279472 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚48471098572374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18279470 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚49828585943552 _step‚218 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47336424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29047282 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18279472 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚47230307715072 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2420263419904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29047282 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47326752 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47230290675712 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18716645 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47342602651648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49650554095616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47230290675712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2414425645818 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.785804938856231e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚49650658571776 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18279470 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2420263419904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚1128.129807475 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47336424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2414339842582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10833084 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚49828743611392 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47342627555328 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46056775769152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2420350856704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29056952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7883809 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2485983291904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29056952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47326752 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2486116056064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251WÊ&¿ÄM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚49650658571776 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18279470 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2420263419904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚1128.129807475 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47336424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2414339842582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10833084 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚49828743611392 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47342627555328 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46056775769152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2420350856704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29056952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7883809 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2485983291904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29056952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47326752 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2486116056064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚47230307715072 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18279472 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚48471098572374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18279470 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚49828585943552 _step‚218 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47336424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29047282 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18279472 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚47230307715072 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2420263419904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29047282 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47326752 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47230290675712 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18716645 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47342602651648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49650554095616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47230290675712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2414425645818 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.785804938856231e+09Ú;Ƥ 2À- _wandbruntime‚1128  _runtime‚1128²âgÎÁþ 2loss/SRqqt1/assignment_loss‚1.4719844824419572 /loss/FBt1/detection_loss‚4.8168214262441325 /loss/FBt2/detection_loss‚3.6138012417520353 0loss/FRt2/assignment_loss‚3.8894664578967624 0loss/FBt1/assignment_loss‚0.0760575647236692 /loss/FRt2/detection_loss‚3.7988653533042425 1loss/SRqqt1/detection_loss‚1.7621778783213633 &loss/total_loss‚31.079355335166493 epoch‚0 /loss/FRt1/assignment_loss‚3.975834024086427 1loss/FBt2/assignment_loss‚0.20706174338203773 0loss/SRqqt2/detection_loss‚2.828880607532723 % _timestamp‚1.7858049388577952e+09 _step‚219 1loss/SRqqt2/assignment_loss‚1.764215966235056 /loss/FRt1/detection_loss‚2.8741885892460854 trainer/global_step‚2749 _runtime‚1128.130891757„Þ  ƒ 1loss/SRqqt2/assignment_loss‚1.764215966235056 /loss/FRt1/detection_loss‚2.8741885892460854 trainer/global_step‚2749 _runtime‚1128.130891757 2loss/SRqqt1/assignment_loss‚1.4719844824419572 /loss/FBt1/detection_loss‚4.8168214262441325 /loss/FBt2/detection_loss‚3.6138012417520353 0loss/FRt2/assignment_loss‚3.8894664578967624 0loss/FBt1/assignment_loss‚0.0760575647236692 /loss/FRt2/detection_loss‚3.7988653533042425 1loss/SRqqt1/detection_loss‚1.7621778783213633 &loss/total_loss‚31.079355335166493 epoch‚0 /loss/FRt1/assignment_loss‚3.975834024086427 1loss/FBt2/assignment_loss‚0.20706174338203773 0loss/SRqqt2/detection_loss‚2.828880607532723 % _timestamp‚1.7858049388577952e+09 _step‚219Ûõ¥OàŽÃ‚(:ƒ –ñÄÓû±¬Û gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚146.88' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚81.6 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›àœa«Ä‚(:   –ñÄÓã»üÝ% disk./.usageGB‚1228.0306854248047 disk.rbd3.in‚0) proc.memory.availableMB‚ 38526.7734375 disk.sda.out‚ 53.43359375 disk.rbd4.in‚0 disk.dm-1.out‚ 53.43359375 disk.drbd1743.out‚1.203125 disk.rbd0.in‚0 disk.dm-0.in‚ 161.53515625" disk.dm-0.out‚4483.14013671875 disk.rbd5.in‚0 disk.dm-1.in‚0.109375 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 161.53515625' disk.nvme0c0n1.out‚4483.14013671875 disk.sda2.in‚0 disk.nvme0n1.in‚ 161.53515625% disk.nvme0n1.out‚4483.14013671875 proc.cpu.threads‚29 network.sent‚ 2.540938e+06 disk.sda.in‚0.109375 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0$ memory_percent‚41.21280908584595 disk.rbd0.out‚0 disk.rbd5.out‚ 0.4873046875 disk.drbd1743.in‚0# proc.memory.rssMB‚ 11266.2265625 disk.rbd4.out‚ 0.45556640625 network.recv‚201857 disk.sda3.out‚ 53.43359375 disk.rbd2.out‚ 0.61328125 disk.sda1.in‚0 disk.rbd1.out‚ 0.55029296875 cpu‚14.203510118131025) disk./.usagePercent‚82.41416968523444 disk.sda3.in‚0.109375 disk.rbd2.in‚0* proc.memory.percent‚17.190897464752197WH˜2Å- _wandbruntime‚1147  _runtime‚1147™útÇÆÁ % _timestamp‚1.7858049578681908e+09 _runtime‚1147.141594948 _step‚220 & lr-AdamW/pg1‚7.213634997237176e-05 & lr-AdamW/pg2‚7.213634997237176e-05 trainer/global_step‚2799qÃð™ÌÇÆ _step‚220 & lr-AdamW/pg1‚7.213634997237176e-05 & lr-AdamW/pg2‚7.213634997237176e-05 trainer/global_step‚2799 % _timestamp‚1.7858049578681908e+09 _runtime‚1147.141594948ÜÕdæ2È- _wandbruntime‚1147  _runtime‚1147-™³®­É¶O C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48169738 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2463379935744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2463379935744 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29564761 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚50534946467840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚49334378006340 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚46877027173472 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚50534841998848 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48071479102464 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚48071462063104 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8024055 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚50534946467840 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚48071462063104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2457436635992 _runtime‚1147.146585572 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18604977 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2457350832868 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 % _timestamp‚1.7858049578692942e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2463467365376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19049812 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11025997 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18604979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚48071479102464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2530224431104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29564761 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50716135578112 )®Ætrainer/global_step‚2799 _step‚221 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18604979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2530359299584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48185751374848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48179396 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18604977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2463467365376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11025995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19050052 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50715975805952 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29564761 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚50534946467840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚49334378006340 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚46877027173472 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚50534841998848 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48071479102464 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚48071462063104 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8024055 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚50534946467840 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚48071462063104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2457436635992 _runtime‚1147.146585572 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18604977 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2457350832868 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 % _timestamp‚1.7858049578692942e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2463467365376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19049812 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11025997 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18604979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚48071479102464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2530224431104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29564761 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50716135578112 trainer/global_step‚2799 _step‚221 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18604979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2530359299584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48185751374848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48179396 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18604977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2463467365376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11025995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19050052 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50715975805952 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48196576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19056569 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29584954 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50734088804864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2464346992128 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2464346992128 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29575284 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2464259555328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2531265174528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚50552992857600 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚48202958491648 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 % _timestamp‚1.7858049582051897e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8026885 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46893784385152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚50552888381440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29584954 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19056819 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18611620 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 trainer/global_step‚2799 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚48088628826112 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48186904 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚1147.478920563 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8026637 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2531130313216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚48088645865472 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚48202983395328 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2458228306090 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0ÊἬÍÅM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2464259555328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚48088645865472 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚49351995651882 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29575284 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11029934 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 _step‚222 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18611622 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50734248569856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2458314109326 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2464346992128 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2464346992128 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29575284 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2464259555328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2531265174528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚50552992857600 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚48202958491648 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 % _timestamp‚1.7858049582051897e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8026885 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46893784385152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚50552888381440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29584954 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19056819 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18611620 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 trainer/global_step‚2799 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚48088628826112 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48186904 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚1147.478920563 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8026637 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2531130313216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚48088645865472 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚48202983395328 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2458228306090 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48196576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19056569 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29584954 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50734088804864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚50215377928332 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8167949 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30102419 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48929817252864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚48929817252864 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚50215275085848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2507463500800 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51621680244736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚47714035789472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2507376071168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18937127 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚51437280753664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2507463500800 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚51437280753664 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚51437176284672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18937129 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51437176284672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2501325099500 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18937129 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚51621518375424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2501239296376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49046107214848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19390796 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚48929800213504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30102419 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8167682 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49046132118528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2575548126208 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30092763 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _runtime‚1167.383574474 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30092763 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936Nñ®òÁ'Ø»O ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2507376071168 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚50215377928332 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8167949 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30102419 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48929817252864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚48929817252864 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚50215275085848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2507463500800 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51621680244736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚47714035789472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2507376071168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18937127 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚51437280753664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2507463500800 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚51437280753664 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚51437176284672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18937129 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51437176284672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2501325099500 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18937129 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚51621518375424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2501239296376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49046107214848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19390796 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚48929800213504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30102419 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8167682 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49046132118528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2575548126208 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30092763 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _runtime‚1167.383574474 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30092763 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936áDô'2Ù- _wandbruntime‚1167  _runtime‚1167"; KÑÚ¿M ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30103286 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚51455327143424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚51639786349056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2576312252416 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚48946966976512 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18943772 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2508343127552 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2502116769598 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚51455222667264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18943772 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30112956 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚51455327143424 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30103286 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30112956 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11226782 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2576447113728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2508343127552 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2508255690752 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 I@Dev9²Ï†ô iceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2502202572834 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 _runtime‚1167.746135979 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49047056 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19397267 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11226784 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49056728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49056728 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251z/9Ê&ÛÄM FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2502202572834 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 _runtime‚1167.746135979 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49047056 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19397267 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11226784 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49056728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49056728 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30103286 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚51455327143424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚51639786349056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2576312252416 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚48946966976512 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18943772 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2508343127552 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2502116769598 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚51455222667264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18943772 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30112956 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚51455327143424 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30103286 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30112956 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11226782 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2576447113728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2508343127552 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2508255690752 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2âØmºø2Ü- _wandbruntime‚1167  _runtime‚1167w'T#Ýý 1loss/FBt1/assignment_loss‚0.11114984542239524 /loss/FBt2/detection_loss‚10.329356419748091 epoch‚0 trainer/global_step‚2849 0loss/FRt2/assignment_loss‚4.0101235846883245 2loss/SRqqt1/assignment_loss‚1.5057342505847762 2loss/SRqqt2/assignment_loss‚1.8382342487573624 1loss/FBt2/assignment_loss‚0.22048474024690223 /loss/FRt1/detection_loss‚3.1833768737316133 % _timestamp‚1.7858049784737606e+09 0loss/FRt1/assignment_loss‚4.0231541220347085 .loss/FRt2/detection_loss‚2.626364669695343 /loss/SRqqt1/detection_loss‚1.53211559288144 1loss/SRqqt2/detection_loss‚2.0061370484541983 /loss/FBt1/detection_loss‚12.459343279989469 %loss/total_loss‚43.84557467623462 _runtime‚ 1167.74717683 _step‚227ØœMÞ‚ 0loss/FRt2/assignment_loss‚4.0101235846883245 2loss/SRqqt1/assignment_loss‚1.5057342505847762 2loss/SRqqt2/assignment_loss‚1.8382342487573624 1loss/FBt2/assignment_loss‚0.22048474024690223 /loss/FRt1/detection_loss‚3.1833768737316133 % _timestamp‚1.7858049784737606e+09 0loss/FRt1/assignment_loss‚4.0231541220347085 .loss/FRt2/detection_loss‚2.626364669695343 /loss/SRqqt1/detection_loss‚1.53211559288144 1loss/SRqqt2/detection_loss‚2.0061370484541983 /loss/FBt1/detection_loss‚12.459343279989469 %loss/total_loss‚43.84557467623462 _runtime‚ 1167.74717683 _step‚227 1loss/FBt1/assignment_loss‚0.11114984542239524 /loss/FBt2/detection_loss‚10.329356419748091 epoch‚0 trainer/global_step‚2849ã íʜ߂(:‘ ´ñÄÓê¶ÅÜ gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚145.933' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚81.07388888888889 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 hÔ¦à‚(:›  ´ñÄÓòéÌß disk.rbd2.out‚ 0.61328125 disk.sda2.out‚0 disk.rbd0.out‚0$ memory_percent‚41.21332764625549 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda3.out‚ 55.04296875 disk.sda.in‚0.109375* proc.memory.availableMB‚38526.43359375 cpu‚14.199759931215349 disk.nvme0n1.in‚ 161.53515625 disk.dm-0.in‚ 161.53515625 disk.rbd4.out‚ 0.51806640625 disk.dm-1.in‚0.109375 proc.cpu.threads‚29) disk./.usagePercent‚82.41485552941866 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.dm-1.out‚ 55.04296875# proc.memory.rssMB‚ 11266.2265625 network.sent‚ 2.630162e+06 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 161.53515625 disk.sda3.in‚0.109375 disk.rbd5.out‚ 0.55029296875* proc.memory.percent‚17.190897464752197% disk./.usageGB‚1228.0409049987793 disk.drbd1743.out‚ 1.23046875 disk.dm-0.out‚4494.169921875 disk.sda2.in‚0% disk.nvme0c0n1.out‚4494.169921875 disk.rbd1.out‚ 0.61328125 disk.sda.out‚ 55.04296875 disk.rbd5.in‚0 network.recv‚206295 disk.drbd1743.in‚0 disk.sda1.out‚0# disk.nvme0n1.out‚4494.169921875[»D#›á‚(: ÃñÄÓó¨ëÜ gpu.0.gpu‚3.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚146.471' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚81.37277777777778 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-MÇ@ â‚(:•  ÃñÄÓõÀß disk.dm-1.in‚0.109375 disk.dm-1.out‚ 55.80859375 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 55.80859375% memory_percent‚41.213345527648926* proc.memory.percent‚17.190897464752197 proc.cpu.threads‚29 disk.rbd2.out‚ 0.61328125 disk.sda2.in‚0% disk.nvme0c0n1.out‚4494.482421875 disk.sda3.in‚0.109375( proc.memory.availableMB‚ 38526.421875 disk.rbd1.out‚ 0.61328125 disk.nvme0n1.in‚ 161.53515625# disk.nvme0n1.out‚4494.482421875 cpu‚14.197728400728174 network.sent‚ 2.674911e+06 network.recv‚208596 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 161.53515625 disk.dm-0.in‚ 161.53515625 disk.rbd4.in‚0 disk.rbd5.in‚0 disk.rbd5.out‚ 0.55029296875 disk.drbd1743.out‚1.234375 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda3.out‚ 55.80859375 disk.rbd0.out‚0# proc.memory.rssMB‚ 11266.2265625$ disk./.usageGB‚1228.040885925293 disk.rbd4.out‚ 0.55029296875' disk./.usagePercent‚82.414854249381 disk.sda.in‚0.109375 disk.drbd1743.in‚0 disk.rbd2.in‚0 disk.dm-0.out‚4494.482421875Å‚Ê|2ã- _wandbruntime‚1187  _runtime‚1187c­qÇäÁ & lr-AdamW/pg1‚7.471356862090237e-05 & lr-AdamW/pg2‚7.471356862090237e-05 trainer/global_step‚2899 % _timestamp‚1.7858049981854036e+09 _runtime‚1187.458707497 _step‚228.:OdÌåÆ trainer/global_step‚2899 % _timestamp‚1.7858049981854036e+09 _runtime‚1187.458707497 _step‚228 & lr-AdamW/pg1‚7.471356862090237e-05 & lr-AdamW/pg2‚7.471356862090237e-05äYa£2æ- _wandbruntime‚1187  _runtime‚1187±˜§Ò»'çµO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚49788138363904 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2551459636224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30630421 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2545213563008 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8311677 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19269279 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚52339510570496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19269277 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚49788155403264 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2551459636224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2620595485696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚51096172165356 _runtime‚1187.463576272 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚228 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49906487958528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 $ _timestamp‚1.785804998186378e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2551372206592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2620719868416 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _step‚229 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚49788138363904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30620765 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚52339615039488 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52339615039488 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8311905 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51096275007840 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124382720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11419695 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19269279 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19731372 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2551372206592 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30620765 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52339510570496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19269277 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49899700 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚52527207826944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149286400 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚48551061444832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2545127759884!ŸDmèºO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19269277 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49899700 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚52527207826944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149286400 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚48551061444832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2545127759884 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚49788138363904 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2551459636224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30630421 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2545213563008 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8311677 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19269279 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚52339510570496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19269277 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚49788155403264 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2551459636224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2620595485696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚51096172165356 _runtime‚1187.463576272 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QIDeviceStatsMoÊXoSnitor.on_train_batch_start/inactive_split.small_pool.current‚228 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49906487958528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 $ _timestamp‚1.785804998186378e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2551372206592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2620719868416 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _step‚229 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚49788138363904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30620765 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚52339615039488 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52339615039488 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8311905 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51096275007840 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124382720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11419695 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19269279 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19731372 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2551372206592 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30620765 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52339510570496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130å¼·ì2é- _wandbruntime‚1187  _runtime‚1187».iÆ&êÀM ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚48567801617152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2546005233106 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19275922 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30631288 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚49805305126912 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52357661429248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚52357661429248 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚49805305126912 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19275920 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚49805322166272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2546091036342 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30640958 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8314536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚49805322166272 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2621502629376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19738419 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52545330468864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49907208 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11423632 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2552251826176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49916880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19275922 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2899 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49907208 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30631288 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19738168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251±‰eË&ëÅM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19738168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚48567801617152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2546005233106 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19275922 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30631288 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚49805305126912 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52357661429248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚52357661429248 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚49805305126912 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19275920 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚49805322166272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2546091036342 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30640958 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8314536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚49805322166272 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2621502629376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19738419 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52545330468864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49907208 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11423632 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2552251826176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49916880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19275922 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2899 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49907208 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30631288 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50750194 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19601427 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2595368342016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2595455771648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 trainer/global_step‚2949 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19601429 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20071753 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53241844856320 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31148767 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19601429 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.785805018726699e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocaé´÷›ted_bytes.small_pool.freed‚ 2595368342016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31158423 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2665802629120 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚51977069244864 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31158423 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚1208.003564776 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚50766843798528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2595455771648 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31148767 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11616547 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2589016223392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚53241949325312 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2665939594752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53241844856320 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11616545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚49388053021472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20071499 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50766818894848 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50759852 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53432783393280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19601427 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚50646493553664 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚49388070060832 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50750194 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19601427 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2595368342016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2595455771648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 trainer/global_step‚2949 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19601429 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20071753 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53241844856320 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31148767 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19601429 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.785805018726699e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2595368342016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31158423 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2665802629120 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚51977069244864 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31158423 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚1208.003564776 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚50766843798528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2595455771648 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31148767 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11616547 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2589016223392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚53241949325312 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2665939594752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53241844856320 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11616545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚49388053021472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20071499 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50766818894848 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130é೎2÷- _wandbruntime‚1208  _runtime‚1208Iœ Æ&øÀM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50784026011648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50777032 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2589893696614 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚53259891238912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51994686890406 trainer/global_step‚2949 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚49404793193792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31159290 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20078272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚50663660316672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31168960 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8458036 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31168960 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚53259995715072 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2596335398400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚49404810233152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50777032 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31159290 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚53450723316224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚53259995715072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50767360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141152768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2666838457344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚50767360 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19608072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2596335398400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50784050915328 % _timestamp‚1.7858050190634584e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11620484 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19608072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2589979499850 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11620482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚53450889372672 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51994789733002 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19608070 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19608070 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8457790 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2666697304576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 _step‚234 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚50663643277312 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0¤ì ŠùÅM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚50767360 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19608072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2596335398400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50784050915328 % _timestamp‚1.7858050190634584e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11620484 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19608072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2589979499850 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11620482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚53450889372672 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51994789733002 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19608070 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19608070 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8457790 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2666697304576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 _step‚234 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚50663643277312 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50784026011648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 G:DeviceStatsMonitor.on_train_batch_endGÉìˆA/active.all.allocated‚50777032 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2589893696614 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚53259891238912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51994686890406 trainer/global_step‚2949 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚49404793193792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31159290 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20078272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚50663660316672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31168960 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8458036 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31168960 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚53259995715072 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2596335398400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚49404810233152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50777032 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31159290 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚53450723316224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚53259995715072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50767360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141152768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2666838457344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200êbáÿð2ú- _wandbruntime‚1208  _runtime‚1208ºáCûû 1loss/SRqqt2/assignment_loss‚1.627494607177196 0loss/FBt1/assignment_loss‚0.2245089772317442 0loss/FBt2/assignment_loss‚0.2906496968832168 /loss/FRt1/detection_loss‚2.9938847988466675 1loss/SRqqt1/detection_loss‚1.6529201839668328 epoch‚0 trainer/global_step‚2949 _runtime‚1208.338058307 0loss/FRt2/assignment_loss‚3.9631897668505824 2loss/SRqqt1/assignment_loss‚1.3717740615750225 0loss/SRqqt2/detection_loss‚2.565632849321539 /loss/FBt1/detection_loss‚2.6807341705505676 _step‚235 .loss/FBt2/detection_loss‚3.498933931463398 /loss/FRt1/assignment_loss‚4.067649871488161 /loss/FRt2/detection_loss‚2.7649008203731027 %loss/total_loss‚27.70227373572803 % _timestamp‚1.7858050190646923e+09–$øü€ /loss/FRt1/detection_loss‚2.9938847988466675 1loss/SRqqt1/detection_loss‚1.6529201839668328 epoch‚0 trainer/global_step‚2949 _runtime‚1208.338058307 0loss/FRt2/assignment_loss‚3.9631897668505824 2loss/SRqqt1/assignment_loss‚1.3717740615750225 0loss/SRqqt2/detection_loss‚2.565632849321539 /loss/FBt1/detection_loss‚2.6807341705505676 _step‚235 .loss/FBt2/detection_loss‚3.498933931463398 /loss/FRt1/assignment_loss‚4.067649871488161 /loss/FRt2/detection_loss‚2.7649008203731027 %loss/total_loss‚27.70227373572803 % _timestamp‚1.7858050190646923e+09 1loss/SRqqt2/assignment_loss‚1.627494607177196 0loss/FBt1/assignment_loss‚0.2245089772317442 0loss/FBt2/assignment_loss‚0.2906496968832168ëÃâz@›ý‚(: áñÄÓùÑÛ gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚119.48' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚66.37777777777778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ig¹˜ þ‚(:•  áñÄÓÙÔ ß disk.nvme0n1.in‚ 161.53515625 disk.rbd0.in‚0 disk.dm-0.in‚ 161.53515625 disk.sda2.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚4495.19189453125 network.recv‚212064) disk./.usagePercent‚82.41486960983305 disk.rbd4.out‚ 0.61328125 disk.dm-1.out‚ 57.13671875 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0* proc.memory.percent‚17.190897464752197$ disk./.usageGB‚1228.041114807129 disk.sda.in‚0.109375 disk.rbd5.in‚0 disk.drbd1743.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0.109375 disk.sda3.out‚ 57.13671875 cpu‚14.193521693954484 network.sent‚ 2.722934e+06 disk.drbd1743.out‚1.28125 disk.sda1.out‚0( proc.memory.availableMB‚ 38526.078125 proc.cpu.threads‚29 disk.rbd3.out‚0' disk.nvme0c0n1.out‚4495.19189453125 disk.sda.out‚ 57.13671875 disk.rbd4.in‚0 disk.rbd5.out‚ 0.61328125 disk.rbd3.in‚0$ memory_percent‚41.21387004852295 disk.dm-1.in‚0.109375 disk.rbd1.out‚ 0.61328125# proc.memory.rssMB‚ 11266.2265625" disk.dm-0.out‚4495.19189453125 disk.rbd2.out‚ 0.61328125" disk.nvme0c0n1.in‚ 161.53515625DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54144179142144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚54338144270336 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 1231.91533047 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31676769 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11813397 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2632990490024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚54144179142144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51620004 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2639364477440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 _step‚237 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19933579 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8597952 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8597709 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚51627174734848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚54144283611136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11813395 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚54144283611136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2711102306816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2639364477440 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50225078676832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2710969535488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚51627199638528 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚50225061637472 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 % _timestamp‚1.7858050426401079e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19933577 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19933577 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2632904686900 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2639451907072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51504814664704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19933579 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31676769 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52857966324372 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2999 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31686425 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 _step‚237 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19933579 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8597952 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8597709 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚51627174734848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚54144283611136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11813395 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚54144283611136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2711102306816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2639364477440 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50225078676832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2710969535488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚51627199638528 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚50225061637472 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 % _timestamp‚1.7858050426401079e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19933577 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19933577 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2632904686900 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2639451907072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51504814664704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19933579 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31676769 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52857966324372 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2999 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31686425 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54144179142144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚54338144270336 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 1231.91533047 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31676769 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11813397 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2632990490024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚54144179142144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51620004íÍx,)2‡ - _wandbruntime‚1232  _runtime‚1232UT‰· ˆ ÀM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51637184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2640331533824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚54356254320128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚52875583969914 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 trainer/global_step‚2999 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7858050429721177e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31696962 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚51644406755328Ƀ¶ ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚51521998467072 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19940220 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31696962 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8600563 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚51644381851648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2712013621248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19940220 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51627512 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51521981427712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20417895 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50241801809792 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _runtime‚1232.245783143 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚54356420376576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51627512 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2640331533824 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141152768 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 _step‚238 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51521998467072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19940222 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20418148 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51637184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2633867963358 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚54162225524736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2640244097024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8600814 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11817332 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2633782160122 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51637184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2633867963358 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚54162225524736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2640244097024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8600814 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11817332 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2633782160122 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2640331533824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚54356254320128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚52875583969914 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 trainer/global_step‚2999 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7858050429721177e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31696962 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚51644406755328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚51521998467072 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19940220 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31696962 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8600563 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚51644381851648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2712013621248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19940220 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51627512 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51521981427712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20417895 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50241801809792 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _runtime‚1232.245783143 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚54356420376576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51627512 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2640331533824 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141152768 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 _step‚238 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51521998467072îOŒ`W2Š - _wandbruntime‚1232  _runtime‚1232©<¨‹ û /loss/FRt1/assignment_loss‚4.179856119021563 1loss/SRqqt1/assignment_loss‚1.354355512944571 2loss/SRqqt2/assignment_loss‚1.5693292664881382 .loss/FBt2/detection_loss‚4.286744770833557 % _timestamp‚1.7858050429733634e+09 1loss/FBt2/assignment_loss‚0.17867106619370623 .loss/FBt1/detection_loss‚2.009713447163692 .loss/FRt2/assignment_loss‚4.14336893368851 /loss/FRt1/detection_loss‚3.2221153613547204 1loss/SRqqt1/detection_loss‚1.4520148687027095 0loss/SRqqt2/detection_loss‚2.738411061880379 &loss/total_loss‚27.910308201803545 _runtime‚1232.246499385 1loss/FBt1/assignment_loss‚0.03376509058712558 /loss/FRt2/detection_loss‚2.7419627029448748 epoch‚0 trainer/global_step‚2999 _step‚239)ñ6ÔŒ € 1loss/FBt1/assignment_loss‚0.03376509058712558 /loss/FRt2/detection_loss‚2.7419627029448748 epoch‚0 trainer/global_step‚2999 _step‚239 /loss/FRt1/assignment_loss‚4.179856119021563 1loss/SRqqt1/assignment_loss‚1.354355512944571 2loss/SRqqt2/assignment_loss‚1.5693292664881382 .loss/FBt2/detection_loss‚4.286744770833557 % _timestamp‚1.7858050429733634e+09 1loss/FBt2/assignment_loss‚0.17867106619370623 .loss/FBt1/detection_loss‚2.009713447163692 .loss/FRt2/assignment_loss‚4.14336893368851 /loss/FRt1/detection_loss‚3.2221153613547204 1loss/SRqqt1/detection_loss‚1.4520148687027095 0loss/SRqqt2/detection_loss‚2.738411061880379 &loss/total_loss‚27.910308201803545 _runtime‚1232.246499385ï”ZN œ ‚(:‘ ÿñÄÓ„¥àÛ gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚126.762' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚70.42333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ)Gf Ž ‚(:•  ÿñÄÓá Ñß$ disk.nvme0n1.out‚4496.6982421875 disk.rbd0.out‚0# proc.memory.rssMB‚ 11266.2265625* proc.memory.percent‚17.190897464752197 disk.sda1.out‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 161.53515625 disk.rbd1.out‚ 0.61328125 disk.rbd0.in‚0$ memory_percent‚41.21394753456116! disk.dm-0.out‚4496.6982421875 disk.dm-1.out‚ 58.4453125 disk.sda2.out‚0& disk.nvme0c0n1.out‚4496.6982421875 disk.rbd1.in‚0 network.recv‚215455) disk./.usagePercent‚82.41486884181045 disk.rbd2.out‚ 0.61328125 disk.rbd3.in‚0 disk.nvme0n1.in‚ 161.53515625 disk.dm-0.in‚ 161.53515625 disk.rbd5.in‚0 disk.drbd1743.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0.109375 disk.rbd4.in‚0 disk.rbd4.out‚ 0.67626953125 disk.dm-1.in‚0.109375 disk.drbd1743.out‚1.296875 disk.sda3.out‚ 58.4453125 proc.cpu.threads‚29$ disk./.usageGB‚1228.041103363037 disk.sda.in‚0.109375 disk.rbd5.out‚ 0.61328125* proc.memory.availableMB‚38526.02734375 cpu‚14.185781530324062 network.sent‚ 2.770307e+06 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 58.4453125ã« 2 - _wandbruntime‚1251  _runtime‚1251àà%Å ¿ & lr-AdamW/pg1‚7.857939659369829e-05 & lr-AdamW/pg2‚7.857939659369829e-05 trainer/global_step‚3049 # _timestamp‚1.78580506241138e+09 _runtime‚1251.686573003 _step‚240,ó¿Ê‘ Ä _runtime‚1251.686573003 _step‚240 & lr-AdamW/pg1‚7.857939659369829e-05 & lr-AdamW/pg2‚7.857939659369829e-05 trainer/global_step‚3049 # _timestamp‚1.78580506241138e+09ðƒÿø2’ - _wandbruntime‚1251  _runtime‚1251hÜ:ö»'“ µO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2683360612864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20750327 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8740082 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 trainer/global_step‚3049 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚52487530574848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51062070253472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2676793150408 $ _timestamp‚1.785805062412061e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32214427 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚53738863403880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚53738966246364 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55046513427968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55243622098944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2676878953532 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51062087292832 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚52363169854464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2683448042496 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55046617896960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20265727 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32204771 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2683448042496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52480156 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55243786065408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20265729 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52470498 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12010245 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚52363152815104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2683360612864 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚53738966246364 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55046513427968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55243622098944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2676878953532 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51062087292832 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚52363169854464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2683448042496 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55046617896960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20265727 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32204771 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2683448042496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52480156 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55243786065408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20265729 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52470498 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12010245 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚52363152815104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2683360612864 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52487664 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3049 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52497336 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32224964 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20757334 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8742890 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53756583892018 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20272370 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2684240232448 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚52380319578112 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚52380336617472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51078810425792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55064664286720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2677756426866 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20272370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2684327669248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚53756481049422 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8743150 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52487664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2756995358208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12014182 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32215294 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12014184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2684327669248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52497336 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚52380336617472 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚52380319578112 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20757072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52504762595328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2757134413824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚1252.021630548 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52487664 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3049 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52497336 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32224964 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20757334 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8742890 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53756583892018 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20272370 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2684240232448 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚52380319578112 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚52380336617472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51078810425792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55064664286720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2677756426866 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20272370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2684327669248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚53756481049422 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8743150 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52487664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2756995358208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12014182 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32215294 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12014184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2684327669248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52497336 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚52380336617472 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚52380319578112 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20757072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52504762595328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2757134413824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0òªyª2˜ - _wandbruntime‚1252  _runtime‚1252-ùšò™ ý 2loss/SRqqt1/assignment_loss‚1.2652488814783283 0loss/SRqqt2/detection_loss‚2.207063931739539 &loss/total_loss‚30.497364604140852 epoch‚0 0loss/FRt1/assignment_loss‚3.8180332379798365 0loss/FBt1/assignment_loss‚0.0623109548268358 0loss/FBt2/assignment_loss‚0.3014589171662347 /loss/FRt1/detection_loss‚3.1345176958057976 /loss/FRt2/detection_loss‚2.7423812273357595 .loss/FBt1/detection_loss‚4.456910012894437 .loss/FBt2/detection_loss‚5.398280725434974 % _timestamp‚1.7858050627496731e+09 2loss/SRqqt2/assignment_loss‚1.8508629693951406 1loss/SRqqt1/detection_loss‚1.4477974876912365 trainer/global_step‚3049 _runtime‚1252.022778059 0loss/FRt2/assignment_loss‚3.8124985623927343 _step‚243‰x™‘š ‚ 2loss/SRqqt1/assignment_loss‚1.2652488814783283 0loss/SRqqt2/detection_loss‚2.207063931739539 &loss/total_loss‚30.497364604140852 epoch‚0 0loss/FRt1/assignment_loss‚3.8180332379798365 0loss/FBt1/assignment_loss‚0.0623109548268358 0loss/FBt2/assignment_loss‚0.3014589171662347 /loss/FRt1/detection_loss‚3.1345176958057976 /loss/FRt2/detection_loss‚2.7423812273357595 .loss/FBt1/detection_loss‚4.456910012894437 .loss/FBt2/detection_loss‚5.398280725434974 % _timestamp‚1.7858050627496731e+09 2loss/SRqqt2/assignment_loss‚1.8508629693951406 1loss/SRqqt1/detection_loss‚1.4477974876912365 trainer/global_step‚3049 _runtime‚1252.022778059 0loss/FRt2/assignment_loss‚3.8124985623927343 _step‚243óªhƒœ› ‚(:‘ ŽòÄÓê´¶Û gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚119.721' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚66.51166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡Çšœ ‚(:  ŽòÄÓÒ©‰à disk.rbd2.out‚ 0.61328125 disk.sda1.in‚0 disk.rbd3.in‚0 network.recv‚217753 disk.rbd1.out‚ 0.61328125 disk.rbd1.in‚0 disk.rbd5.in‚0 disk.dm-1.out‚59.03125 disk.drbd1743.in‚0' proc.memory.availableMB‚ 38526.21875" disk.nvme0c0n1.in‚ 161.53515625 disk.sda3.out‚59.03125 disk.rbd5.out‚ 0.61328125 disk.drbd1743.out‚ 1.31640625& disk.nvme0c0n1.out‚4496.9326171875 disk.rbd4.in‚0 disk.rbd4.out‚ 0.67626953125 disk.rbd2.in‚0 disk.sda1.out‚0 network.sent‚ 2.815188e+06) disk./.usagePercent‚82.41907197350761$ disk.nvme0n1.out‚4496.9326171875 disk.sda3.in‚0.109375 disk.sda2.in‚0 disk.rbd3.out‚0$ memory_percent‚41.21365547180176% disk./.usageGB‚1228.1037330627441! disk.dm-0.out‚4496.9326171875 disk.sda.in‚0.109375 disk.sda2.out‚0* proc.memory.percent‚17.190897464752197 cpu‚14.183637698104869 proc.cpu.threads‚29 disk.rbd0.in‚0 disk.dm-0.in‚ 161.53515625# proc.memory.rssMB‚ 11266.2265625 disk.nvme0n1.in‚ 161.53515625 disk.rbd0.out‚0 disk.sda.out‚59.03125 disk.dm-1.in‚0.109375vP £2 - _wandbruntime‚1272  _runtime‚1272’³‡'Åž ¿ % lr-AdamW/pg1‚7.98680059179636e-05 % lr-AdamW/pg2‚7.98680059179636e-05 trainer/global_step‚3099 % _timestamp‚1.7858050829114065e+09 _runtime‚1272.186615711 _step‚244¬£àÊŸ Ä _step‚244 % lr-AdamW/pg1‚7.98680059179636e-05 % lr-AdamW/pg2‚7.98680059179636e-05 trainer/global_step‚3099 % _timestamp‚1.7858050829114065e+09 _runtime‚1272.186615711ô2Æݯ2  - _wandbruntime‚1272  _runtime‚1272‘ ¶O SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8882282 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚5614932675“'¢…'6352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚53221490965504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2727444177920 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55948847713792 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2727356748288 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149286400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚53347911318528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20597877 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2801291055104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2727356748288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32742429 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12207097 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20597879 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21089377 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚54619760483388 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20597879 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚53221508004864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55948952182784 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚55948847713792 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚53221508004864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚53347886414848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _runtime‚1272.186756875 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8882519 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32732773 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12207095 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20597877 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2727444177920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21089616;‡°Á'¢ »O L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32732773 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53340308 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2720681613916 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55948952182784 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55948847713792 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2727356748288 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149286400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚53347911318528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20597877 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2801291055104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2727356748288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32742429 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12207097 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20597879 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21089377 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚54619760483388 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20597879 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚53221508004864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55948952182784 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚55948847713792 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚53221508004864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚53347886414848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _runtime‚1272.186756875 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8882519 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32732773 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12207095 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20597877 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2727444177920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21089616 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8882282 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚56149326756352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚53221490965504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2727444177920 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21096050 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20604522 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53238674767872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21096288 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51915819041792 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53238657728512 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55966998572544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚54637378128930 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚1272.558608996 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12211032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7858050832847307e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53238657728512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8885018 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51915836081152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2721644890374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚246 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32752966 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2728236367872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32743296 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53347816 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55966998572544 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53238674767872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53357488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚53365093531648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2802325490688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32743296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 trainer/global_step‚3099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2721559087138 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53347816 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56167286258176 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚54637480971526 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2728236367872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53357488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚53365093531648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2802325490688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32743296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 trainer/global_step‚3099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2721559087138 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53347816 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56167286258176 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚54637480971526 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2728236367872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455˜+î­936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21096050 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20604522 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53238674767872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21096288 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51915819041792 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53238657728512 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55966998572544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚54637378128930 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚1272.558608996 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12211032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7858050832847307e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53238657728512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8885018 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51915836081152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2721644890374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚246 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32752966 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2728236367872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32743296 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548öÚ(ï:2¦ - _wandbruntime‚1272  _runtime‚12724† § ü trainer/global_step‚3099 _runtime‚1272.559335598 /loss/FRt1/detection_loss‚2.8818061402658137 .loss/FRt2/detection_loss‚2.838667732782853 1loss/SRqqt2/detection_loss‚2.0990947956836603 % _timestamp‚1.7858050832861655e+09 /loss/FRt1/assignment_loss‚4.123601951250216 /loss/FRt2/assignment_loss‚3.842731550717965 1loss/SRqqt1/detection_loss‚1.3850285195511287 _step‚247 .loss/FBt1/detection_loss‚4.536073997803297 /loss/FBt2/detection_loss‚4.1701220464892685 epoch‚0 1loss/SRqqt1/assignment_loss‚1.416073919858697 2loss/SRqqt2/assignment_loss‚1.7827211900878308 1loss/FBt1/assignment_loss‚0.16880719650097573 1loss/FBt2/assignment_loss‚0.05544765071268697 %loss/total_loss‚29.30017669170439ø%í¨  1loss/SRqqt1/assignment_loss‚1.416073919858697 2loss/SRqqt2/assignment_loss‚1.7827211900878308 1loss/FBt1/assignment_loss‚0.16880719650097573 1loss/FBt2/assignment_loss‚0.05544765071268697 %loss/total_loss‚29.30017669170439 trainer/global_step‚3099 _runtime‚1272.559335598 /loss/FRt1/detection_loss‚2.8818061402658137 .loss/FRt2/detection_loss‚2.838667732782853 1loss/SRqqt2/detection_loss‚2.0990947956836603 % _timestamp‚1.7858050832861655e+09 /loss/FRt1/assignment_loss‚4.123601951250216 /loss/FRt2/assignment_loss‚3.842731550717965 1loss/SRqqt1/detection_loss‚1.3850285195511287 _step‚247 .loss/FBt1/detection_loss‚4.536073997803297 /loss/FBt2/detection_loss‚4.1701220464892685 epoch‚0÷U ›© ‚(: òÄÓåäÅÛ gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚151.729' gpu.0.enforcedPowerLimitWatts‚180.0' gpu.0.powerPercent‚84.2938888888889 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eÖåÅ£ª ‚(:˜  òÄÓÖ¢‘á disk.rbd2.in‚0* proc.memory.percent‚17.190897464752197" disk.nvme0c0n1.in‚ 161.53515625 disk.rbd0.in‚0! disk.dm-0.out‚4559.6298828125 disk.sda1.out‚0 disk.rbd1.out‚ 0.61328125 disk.sda3.in‚0.109375 disk.rbd5.out‚ 0.61328125 disk.drbd1743.out‚1.34375 network.sent‚ 2.859998e+06) disk./.usagePercent‚82.41905789309321% disk./.usageGB‚1228.1035232543945 disk.sda3.out‚ 59.81640625 disk.dm-0.in‚ 161.53515625 disk.dm-1.out‚ 59.81640625% memory_percent‚41.214072704315186 disk.sda.out‚ 59.81640625 disk.rbd4.in‚0 disk.rbd4.out‚ 0.67626953125 disk.drbd1743.in‚0 disk.sda1.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 11266.2265625& disk.nvme0c0n1.out‚4559.6298828125 disk.rbd2.out‚ 0.61328125 network.recv‚220337 disk.nvme0n1.in‚ 161.53515625$ disk.nvme0n1.out‚4559.6298828125 disk.sda.in‚0.109375 disk.dm-1.in‚0.109375 disk.rbd3.in‚0) proc.memory.availableMB‚ 38525.9453125 disk.rbd0.out‚0 disk.rbd5.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0 cpu‚14.181798639743077 proc.cpu.threads‚29 disk.rbd1.in‚0öÒTÖœ« ‚(:‘ ¬òÄÓ¢âÉÛ gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚138.201' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚76.77833333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`_›|ž¬ ‚(:“  ¬òÄÓä؇á" disk.nvme0c0n1.in‚ 161.53515625 disk.rbd1.in‚0* proc.memory.percent‚17.190897464752197 network.recv‚222546) disk./.usagePercent‚82.41905533301788 disk.sda2.out‚0 disk.rbd1.out‚ 0.61328125 disk.rbd5.out‚ 0.61328125 cpu‚14.180292347685835 disk.rbd3.out‚0 disk.rbd4.in‚0# memory_percent‚41.2137508392334' proc.memory.availableMB‚ 38526.15625% disk./.usageGB‚1228.1034851074219 disk.drbd1743.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 60.66796875 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 11266.2265625 proc.cpu.threads‚29 disk.nvme0n1.in‚ 161.53515625 disk.sda.in‚0.109375 disk.sda2.in‚0 disk.rbd3.in‚0& disk.nvme0c0n1.out‚4559.8798828125 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 60.66796875 disk.drbd1743.out‚1.34375$ disk.nvme0n1.out‚4559.8798828125 disk.sda3.in‚0.109375! disk.dm-0.out‚4559.8798828125 disk.rbd4.out‚ 0.67626953125 disk.rbd5.in‚0 network.sent‚ 2.90476e+06 disk.rbd2.in‚0 disk.rbd2.out‚ 0.61328125 disk.sda1.out‚0 disk.dm-0.in‚ 161.53515625 disk.sda.out‚ 60.66796875ŽÝœ2­ - _wandbruntime‚1292  _runtime‚1292½brÅ® ¿ % lr-AdamW/pg2‚8.11566152422289e-05 trainer/global_step‚3149 % _timestamp‚1.7858051030410933e+09 _runtime‚1292.315614764 _step‚248 % lr-AdamW/pg1‚8.11566152422289e-05DušTʯ Ä % lr-AdamW/pg1‚8.11566152422289e-05 % lr-AdamW/pg2‚8.11566152422289e-05 trainer/global_step‚3149 % _timestamp‚1.7858051030410933e+09 _runtime‚1292.315614764 _step‚248øq;.z2° - _wandbruntime‚1292  _runtime‚1292Œ»tÔ¼'± ¶O LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚1292.315718518 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33270431 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚56851286468608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚57054865377792 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚54079829115904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚54079846155264 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 trainer/global_step‚3149 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2846598219264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21428865 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 _step‚249 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21428605 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54208267158528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2771440313344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚56851286468608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52736104524832 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2764655880548 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33260775 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚54079829115904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20930029 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚57054699314176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54208242254848 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2764570077424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚55500657562896 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2771440313344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2846457059328 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2771352883712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12403947 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9024918 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54190802 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33270431 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚56851181999616 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52736087485472 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20930029 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚56851181999616 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52736087485472 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20930029 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 _step‚249 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21428605 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54208267158528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2771440313344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚56851286468608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52736104524832 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2764655880548 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33260775 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚54079829115904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20930029 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚57054699314176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54208242254848 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2764570077424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G?DeviceStatsMonitor.on_t m2rain_batch_start/inactive_split.all.peak‚548 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚55500657562896 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640ùL·02³ - _wandbruntime‚1292  _runtime‚12929Å(<Æ&´ ÀM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12407882 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33280968 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55518275208438 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52752844697152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21435382 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2847362607616 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2772319940096 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54217640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33271298 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2772319940096 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 _runtime‚1292.651867228 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20936670 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9027500 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2847495371776 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54207968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚54097012918272 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚250 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33271298 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54225449371648 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚55518378051034 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33280968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20936670 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2765533353882 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20936672UÙ’)Ë&µ ÅM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12407884 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21435382 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2847362607616 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2772319940096 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54217640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33271298 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2772319940096 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 _runtime‚1292.651867228 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20936670 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9027500 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2847495371776 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54207968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚54097012918272 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚250 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33271298 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54225449371648 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚55518378051034 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33280968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20936670 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2765533353882 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20936672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12407882 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33280968 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55518275208438 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52752844697152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚54938167266304 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57753516285440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 166063616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚54938184305664 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57753620754432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55068598094848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _runtime‚ 1312.48160684 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57753620754432 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55050954 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚53573113140832 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33788777 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚54938184305664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 L?DeviceStatsMonH’Çkitor.on_train_batch_start/active.large_pool.freed‚21262177 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57753516285440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12600795 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚53573096101472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121c“ÁïÀ'À ºO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21262179 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚56381554642404 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 % _timestamp‚1.7858051232048385e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2815436448768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55060612 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21768390 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚57960293331456 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2891836396544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33798433 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚56381657484888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚3199 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚54938167266304 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57753516285440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 166063616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚54938184305664 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57753620754432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55068598094848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _runtime‚ 1312.48160684 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57753620754432 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55050954 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚53573113140832 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33788777 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚54938184305664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21262177 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57753516285440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12600795 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚53573096101472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12600797 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9167328 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚55068622998528 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21774877 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚53589836273792 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57771562668032 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9170145 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55077792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9170383 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55068120 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57978408804352 % _timestamp‚1.7858051235574362e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21268820 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33799300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21268822 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚54955334029312 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚53589853313152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2816316075520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2816228638720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21775117 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12604732 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 _step‚254 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2816316075520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2892732162560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55077792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55085830115328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55068120 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 trainer/global_step‚3199 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56399275130542 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33799300 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12604734 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚54955351068672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2809336014154 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54955351068672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54955334029312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57771667144192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55077792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55085830115328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55068120 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 trainer/global_step‚3199 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56399275130542 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33799300 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12604734 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚54955351068672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2809336014154 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54955351068672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54955334029312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57771667144192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21774877 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚53589836273792 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57771562668032 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9170145 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55077792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9170383 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55068120 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57978408804352 % _timestamp‚1.7858051235574362e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21268820 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33799300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21268822 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚54955334029312 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚53589853313152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0þ†Á¬2Ä - _wandbruntime‚1312  _runtime‚1312à×…³Å ú 2loss/SRqqt2/assignment_loss‚1.4949693791028382 $loss/total_loss‚31.4880433304194 $ _timestamp‚1.785805123558868e+09 0loss/FRt2/assignment_loss‚3.9604924270292607 1loss/FBt1/assignment_loss‚0.06515731145773315 .loss/FRt2/detection_loss‚2.768085232231675 1loss/SRqqt2/detection_loss‚3.6857644734305794 /loss/FBt1/detection_loss‚3.1801687866460724 .loss/FBt2/detection_loss‚6.269101903632132 _step‚255 /loss/FRt1/assignment_loss‚3.977819868326187 0loss/FBt2/assignment_loss‚0.1562489285894404 /loss/FRt1/detection_loss‚2.5033715850114824 epoch‚0 _runtime‚1312.832017942 2loss/SRqqt1/assignment_loss‚1.4217068145581342 0loss/SRqqt1/detection_loss‚2.005156620403868 trainer/global_step‚3199­¬ê6”Æ ÿ epoch‚0 _runtime‚1312.832017942 2loss/SRqqt1/assignment_loss‚1.4217068145581342 0loss/SRqqt1/detection_loss‚2.005156620403868 trainer/global_step‚3199 2loss/SRqqt2/assignment_loss‚1.4949693791028382 $loss/total_loss‚31.4880433304194 $ _timestamp‚1.785805123558868e+09 0loss/FRt2/assignment_loss‚3.9604924270292607 1loss/FBt1/assignment_loss‚0.065157311457íCœ.q73315 .loss/FRt2/detection_loss‚2.768085232231675 1loss/SRqqt2/detection_loss‚3.6857644734305794 /loss/FBt1/detection_loss‚3.1801687866460724 .loss/FBt2/detection_loss‚6.269101903632132 _step‚255 /loss/FRt1/assignment_loss‚3.977819868326187 0loss/FBt2/assignment_loss‚0.1562489285894404 /loss/FRt1/detection_loss‚2.5033715850114824ÿgô†œÇ ‚(:‘ ÊòÄÓ±¢½Û gpu.0.gpu‚44.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚146.761' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚81.53388888888888 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ³wu¡È ‚(:–  ÊòÄÓôÌŸß disk.sda.in‚0.109375 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 161.53515625 disk.rbd4.out‚ 0.67626953125 disk.dm-1.out‚ 61.8828125 disk.rbd2.out‚ 0.61328125% disk.nvme0c0n1.out‚4622.962890625# proc.memory.rssMB‚ 11266.2265625) disk./.usagePercent‚82.42331529838764 disk.sda3.in‚0.109375 disk.sda.out‚ 61.8828125 disk.sda1.in‚0% memory_percent‚41.214579343795776* proc.memory.percent‚17.190897464752197 cpu‚14.176444875104389 network.sent‚ 2.952168e+06 disk.rbd5.in‚0 disk.drbd1743.in‚0 disk.drbd1743.out‚ 1.37890625 disk.sda2.in‚0 proc.cpu.threads‚29 disk.rbd0.out‚0 disk.dm-0.in‚ 161.53515625 disk.rbd4.in‚0 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.61328125 disk.sda3.out‚ 61.8828125 disk.dm-1.in‚0.109375 disk.sda2.out‚0 disk.nvme0n1.in‚ 161.53515625# disk.nvme0n1.out‚4622.962890625 disk.rbd0.in‚0 disk.dm-0.out‚4622.962890625 disk.rbd5.out‚ 0.61328125 disk.sda1.out‚0 disk.rbd3.in‚0* proc.memory.availableMB‚38525.61328125 network.recv‚225945% disk./.usageGB‚1228.1669616699219ûŽ2É - _wandbruntime‚1332  _runtime‚1332E†ÈÇÊ Á _runtime‚1332.677602179 _step‚256 & lr-AdamW/pg1‚8.373383389075951e-05 & lr-AdamW/pg2‚8.373383389075951e-05 trainer/global_step‚3249 % _timestamp‚1.7858051434023452e+09÷àø)ÌË Æ & lr-AdamW/pg1‚8.373383389075951e-05 & lr-AdamW/pg2‚8.373383389075951e-05 trainer/global_step‚3249 % _timestamp‚1.7858051434023452e+09 _runtime‚1332.677602179 _step‚256€gÔÛ2Ì - _wandbruntime‚1332  _runtime‚1332è禼'Í ¶O E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55911106 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚58655850571264 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2859345154560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚55796505416704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58865829636608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2852432807564 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12797645 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21594327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚58655955040256 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9309394 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34326435 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 trainer/global_step‚3249 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34316779 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚1332.677713628 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34326435 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55928953934848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2937010570240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 _step‚257 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚55796522456064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21594327 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55920764 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58865989408768 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54410121756832¼DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2852432807564 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12797645 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21594327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚58655955040256 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9309394 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34326435 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 trainer/global_step‚3249 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34316779 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚1332.677713628 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34326435 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55928953934848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2937010570240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 _step‚257 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚55796522456064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21594327 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55920764 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58865989408768 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54410121756832 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55911106 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚58655850571264 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2859345154560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚55796505416704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58865829636608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816Ãk-ƒ2Ï - _wandbruntime‚1333  _runtime‚1333$&tÄ_&Ð ÀM ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22113795 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚55813672179712 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2853310280898 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21600970 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34336972 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58883947997184 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21600972 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2860224774144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 trainer/global_step‚3249 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22114041 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2860224774144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2937786945536 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚57280069367454 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7858051437370412e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12801584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2937911321088 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚57280172210050 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21600972 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55937944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9312457 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚55813672179712 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9312213 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _runtime‚1333.010631894 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚58673896953856 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.c÷¹ÆVgurrent‚244 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54426844889792_îÚ/Ë&Ñ ÅM ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22113795 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚55813672179712 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2853310280898 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21600970 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34336972 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58883947997184 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21600972 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2860224774144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 trainer/global_step‚3249 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22114041 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2860224774144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2937786945536 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚57280069367454 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7858051437370412e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12801584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2937911321088 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚57280172210050 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21600972 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55937944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9312457 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚55813672179712 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9312213 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _runtime‚1333.010631894 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚58673896953856 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54426844889792‚DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21926479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚58143348801420 _runtime‚1352.915024864 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2982187654144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22447218 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2903341289984 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59558184857088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21926477 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34854437 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34844781 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2982056979968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21926479 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59771366754816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚55247113333472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2896235467948 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚58143451643904 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640H'QÞ »O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2982187654144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22447218 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2903341289984 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59558184857088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21926477 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34854437 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34844781 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2982056979968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21926479 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59771366754816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚55247113333472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2896235467948 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 HBDeviceStatsMonitor.on_t^Þô¬ rain_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚58143451643904 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12994495 % _timestamp‚1.7858051636370506e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56780916 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22447471 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34854437 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚56789309774848 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21926479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚58143348801420 _runtime‚1352.915024864…þõ»2ß - _wandbruntime‚1353  _runtime‚1353Å÷œ¾Æ&à ÀM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚56672027369472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56798096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2904220909568 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56788424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12998432 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21933120 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22453944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2897198744406 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9455512 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚56806541795328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21933120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21933122 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚56806516891648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55263853505792 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21933122 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 % _timestamp‚1.7858051639842973e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22454194 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 _runtime‚1353.258027453 _step‚262 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚56672010330112 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56798096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚59789634587136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2982960027648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2904308346368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34864974 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2904220909568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚58160966446962 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚59576231239680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34855304 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59576335715840 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56788424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 _runtime‚1353.258027453 _step‚262 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚56672010330112 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56798096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚59789634587136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2982960027648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2904308346368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34864974 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2904220909568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚58160966446962 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚59576231239680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34855304 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59576335715840 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56788424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21933120 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22453944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2897198744406 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9455512 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚56806541795328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21933120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21933122 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚56806516891648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55263853505792 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21933122 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 % _timestamp‚1.7858051639842973e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22454194 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ODeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3027371813888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚57513198756864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60460519142912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2947337425408 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22258627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13191347 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22787057 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22786803 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35382439 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2947337425408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚57649665614848 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9595710 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57631410 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2947424855040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35372783 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2940123931456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22258627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚60460623611904 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22258629 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56084138988832 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚57513181717504 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 % _timestamp‚1.7858051831491082e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2947424855040 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9595458 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 _step‚265 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.D_<'allocated‚60677062332416 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚57513198756864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚60460623611904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22258629 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57513181717504 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59024245880928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2940209734580 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59024348723412 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚1372.422694188 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚60460519142912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640gžÕ¹Á'ì »O _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚57513198756864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60460519142912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2947337425408 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22258627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13191347 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22787057 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22786803 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35382439 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2947337425408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚57649665614848 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9595710 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57631410 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2947424855040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35372783 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2940123931456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22258627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚60460623611904 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22258629 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56084138988832 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚57513181717504 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 % _timestamp‚1.7858051831491082e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2947424855040 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9595458 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 _step‚265 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚60677062332416 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚57513198756864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚60460623611904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22258629 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57513181717504 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59024245880928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2940209734580 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59024348723412 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚1372.422694188 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚60460519142912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35372783 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13191345 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3027371813888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121‰pGÈ2í - _wandbruntime‚1372  _runtime‚1372QXq\Æ&î ÀM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59041966369066 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57658248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22265272 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚57530348480512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35383306 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚3349 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2948304481792 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22793555 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9598528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3028269005312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13195284 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35392976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2941001404678 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57648576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56100879161152 _runtime‚1372.759711996 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22265270 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22793812 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚60695166640640 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚57530365519872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13195282 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57658248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57648576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _step‚266 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2948304481792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60478670001664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚57530365519872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚60478670001664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7858051834861248e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 5@þ!!ï ÅM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚57530365519872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13195282 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57658248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57648576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _step‚266 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2948304481792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60478670001664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚57530365519872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚60478670001664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7858051834861248e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59041966369066 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57658248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22265272 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚57530348480512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35383306 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚3349 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2948304481792 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22793555 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9598528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3028269005312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13195284 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35392976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2941001404678 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_poolF2.|ª.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57648576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56100879161152 _runtime‚1372.759711996 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22265270 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22793812 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚60695166640640 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800Šoä.2ð - _wandbruntime‚1372  _runtime‚1372@VX”þñ ø .loss/FBt1/detection_loss‚3.588955072213139 trainer/global_step‚3349 % _timestamp‚1.7858051834873526e+09 _runtime‚ 1372.76048763 1loss/FBt1/assignment_loss‚0.31448103323509713 epoch‚0 2loss/SRqqt2/assignment_loss‚1.6731760062036976 1loss/FBt2/assignment_loss‚0.17473346758735084 /loss/FRt2/detection_loss‚2.4882934186193677 -loss/SRqqt2/detection_loss‚2.105945928284 /loss/FRt1/assignment_loss‚4.064053377379542 /loss/FBt2/detection_loss‚3.0435343391367033 %loss/total_loss‚26.54407919721809 _step‚267 /loss/FRt2/assignment_loss‚3.816966869433721 2loss/SRqqt1/assignment_loss‚1.3587416116360242 .loss/FRt1/detection_loss‚2.545456441200298 1loss/SRqqt1/detection_loss‚1.3697416322891525 ò ý /loss/FRt2/assignment_loss‚3.816966869433721 2loss/SRqqt1/assignment_loss‚1.3587416116360242 .loss/FRt1/detection_loss‚2.545456441200298 1loss/SRqqt1/detection_loss‚1.3697416322891525 .loss/FBt1/detection_loss‚3.588955072213139 trainer/global_step‚3349 % _timestamp‚1.7858051834873526e+09 _runtime‚ 1372.76048763 1loss/FBt1/assignment_loss‚0.31448103323509713 epoch‚0 2loss/SRqqt2/assignment_loss‚1.6731760062036976 1loss/FBt2/assignment_loss‚0.17473346758735084 /loss/FRt2/detection_loss‚2.4882934186193677 -loss/SRqqt2/detection_loss‚2.105945928284 /loss/FRt1/assignment_loss‚4.064053377379542 /loss/FBt2/detection_loss‚3.0435343391367033 %loss/total_loss‚26.54407919721809 _step‚267‹©Άœó ‚(:‘ †óÄÓ¬ð¥Û gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚150.547' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚83.63722222222222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«±9ô¥ô ‚(:š  †óÄÓ‹Ž¢ß disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 0.61328125 disk.sda2.in‚0 network.sent‚ 3.088656e+06% disk.nvme0n1.out‚4687.39990234375 disk.rbd5.out‚ 0.61328125 disk.sda2.out‚0' disk.nvme0c0n1.out‚4687.39990234375 disk.rbd0.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚29 disk.sda1.in‚0 disk.rbd1.out‚ 0.61328125 disk.sda3.in‚0.109375 disk.rbd4.in‚0 disk.dm-1.out‚ 64.4765625" disk.nvme0c0n1.in‚ 161.53515625 cpu‚14.170742338006393) disk./.usagePercent‚82.43173180208332 disk.sda3.out‚ 64.4765625 disk.sda.out‚ 64.4765625 disk.rbd4.out‚ 0.67626953125 disk.drbd1743.out‚ 1.44140625# proc.memory.rssMB‚ 11266.2265625* proc.memory.percent‚17.190897464752197 network.recv‚233816" disk.dm-0.out‚4687.39990234375 disk.sda1.out‚0) proc.memory.availableMB‚ 38525.7109375 disk.dm-0.in‚ 161.53515625 disk.sda.in‚0.109375 disk.rbd5.in‚0 disk.drbd1743.in‚0% disk./.usageGB‚1228.2923736572266 disk.nvme0n1.in‚ 161.53515625 disk.rbd0.out‚0 disk.dm-1.in‚0.109375 disk.rbd3.in‚0$ memory_percent‚41.21443033218384-×¼2õ - _wandbruntime‚1392  _runtime‚1392hisìÇö Á & lr-AdamW/pg2‚8.759966186355543e-05 trainer/global_step‚3399 % _timestamp‚1.7858052034409587e+09 _runtime‚1392.714305671 _step‚268 & lr-AdamW/pg1‚8.759966186355543e-05IVÇ×Ì÷ Æ trainer/global_step‚3399 % _timestamp‚1.7858052034409587e+09 _runtime‚1392.714305671 _step‚268 & lr-AdamW/pg1‚8.759966186355543e-05 & lr-AdamW/pg2‚8.759966186355543e-05Œ¯#OÂ2ø - _wandbruntime‚1392  _runtime‚1392+<Š¼'ù ¶O IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23126753 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9738297 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58510021454848 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59905245802920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35900785 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59905142960436 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58371519867904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58371536907264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚61582564283904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3072385154048 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2991333560832 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56921130565472 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _runtime‚1392.719600265 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚61362957897728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2991333560832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56921147604832 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _step‚269 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61362853428736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58510046358528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35910441 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23126492 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22590779 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13388197 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3072517925376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58501220 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22590777 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13388195 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61362957897728 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61362853428736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58371536907264 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58371519867904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2984012394964 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2991420990464 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22590777 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ÎOrÁ'ú »O RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2991333560832 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56921130565472 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _runtime‚1392.719600265 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚61362957897728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2991333560832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56921147604832 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _step‚269 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61362853428736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58510046358528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35910441 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23126492 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22590779 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13388197 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3072517925376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58501220 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22590777 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13388195 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61362957897728 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61362853428736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58371536907264 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58371519867904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2984012394964 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2991420990464 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22590777 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23126753 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9738297 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58510021454848 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59905245802920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35900785 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59905142960436 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58371519867904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58371536907264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚61582564283904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3072385154048 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _¶š2û - _wandbruntime‚1393  _runtime‚1393$@ûQÚü ÀM _runtime‚1393.062368432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56937887777152 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22597420 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚61600671868928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35920978 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚58388686630912 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23133251 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚58388703670272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13392132 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58508728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23133506 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58527228571648 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _step‚270 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22597422 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚61600516298240 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56937870737792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59922863448574 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59922760605978 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚61381004287488 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2992300617216 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚58388686630912 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61380899811328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58518400 F